Skip to main content

Move from dbt Core to dbt Cloud: What you need to know

Updated
Migration
dbt Core
dbt Cloud
Intermediate
Menu

    Introduction

    Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service.

    Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:

    Guide
    Information
    Audience
    Move from dbt Core to dbt Cloud: What you need to knowUnderstand the considerations and methods needed in your move from dbt Core to dbt Cloud.Team leads
    Admins
    Move from dbt Core to dbt Cloud: Get startedLearn the steps needed to move from dbt Core to dbt Cloud.Developers
    Data engineers
    Data analysts
    Move from dbt Core to dbt Cloud: Optimization tipsLearn how to optimize your dbt Cloud experience with common scenarios and useful tips.Everyone

    Why move to dbt Cloud?

    If your team is using dbt Core today, you could be reading this guide because:

    • You’ve realized the burden of maintaining that deployment.
    • The person who set it up has since left.
    • You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices.

    Moving from dbt Core to dbt Cloud simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With dbt Cloud, you gain access to features like cross-team collaboration (dbt Mesh), version management, streamlined CI/CD, dbt Explorer for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently.

    It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.

     What is dbt Cloud and dbt Core?

    What you'll learn

    Today thousands of companies, with data teams ranging in size from 2 to 2,000, rely on dbt Cloud to accelerate data work, increase collaboration, and win the trust of the business. Understanding what you'll need to do in order to move between dbt Cloud and your current Core deployment will help you strategize and plan for your move.

    The guide outlines the following steps:

    • Considerations: Learn about the most important things you need to think about when moving from Core to Cloud.
    • Plan your move: Considerations you need to make, such as user roles and permissions, onboarding order, current workflows, and more.
    • Move to dbt Cloud: Review the steps to move your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository.
    • Test and validate: Discover how to ensure model accuracy and performance post-move.
    • Transition and training: Learn how to fully transition to dbt Cloud and what training and support you may need.
    • Summary: Summarizes key takeaways and what you've learned in this guide.
    • What's next?: Introduces what to expect in the following guides.

    Considerations

    If your team is using dbt Core today, you could be reading this guide because:

    • You’ve realized the burden of maintaining that deployment.
    • The person who set it up has since left.
    • You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices.

    This guide shares the technical adjustments and team collaboration strategies you’ll need to know to move your project from dbt Core to dbt Cloud. Each "build your own" deployment of dbt Core will look a little different, but after seeing hundreds of teams make the migration, there are many things in common.

    The most important things you need to think about when moving from dbt Core to dbt Cloud:

    • How is your team structured? Are there natural divisions of domain?
    • Should you have one project or multiple? Which dbt resources do you want to standardize & keep central?
    • Who should have permission to view, develop, and administer?
    • How are you scheduling your dbt models to run in production?
    • How are you currently managing Continuous integration/Continuous deployment (CI/CD) of logical changes (if at all)?
    • How do your data developers prefer to work?
    • How do you manage different data environments and the different behaviors in those environments?

    dbt Cloud provides standard mechanisms for tackling these considerations, all of which deliver long-term benefits to your organization:

    • Cross-team collaboration
    • Access control
    • Orchestration
    • Isolated data environments

    If you have rolled out your own dbt Core deployment, you have probably come up with different answers.

    Plan your move

    As you plan your move, consider your workflow and team layout to ensure a smooth transition. Here are some key considerations to keep in mind:

     Start small to minimize risk and maximize learning
     User roles and responsibilities
     Onboarding order
     Analyze current workflows, review processes, and team structures
     Understand access control
     Manage environments

    Move to dbt Cloud

    This guide is your roadmap to help you think about migration strategies and what moving from dbt Core to dbt Cloud could look like.

    After reviewing the considerations and planning your move, you may want to start moving your dbt Core project to dbt Cloud:

    For a more detailed comparison of dbt Core and dbt Cloud, check out How dbt Cloud compares with dbt Core.

    Test and validate

    After setting the foundations of dbt Cloud, it's important to validate your migration to ensure seamless functionality and data integrity:

    • Review your dbt project: Ensure your project compiles correctly and that you can run commands. Make sure your models are accurate and monitor performance post-move.
    • Start cutover: You can start the cutover to dbt Cloud by creating a dbt Cloud job with commands that only run a small subset of the DAG. Validate the tables are being populated in the proper database/schemas as expected. Then continue to expand the scope of the job to include more sections of the DAG as you gain confidence in the results.
    • Precision testing: Use unit testing to allow you to validate your SQL modeling logic on a small set of static inputs before you materialize your full model in production.
    • Access and permissions: Review and adjust access controls and permissions within dbt Cloud to maintain security protocols and safeguard your data.

    Transition and training

    Once you’ve confirmed that dbt Cloud orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you’re still using an external orchestrator (such as Airflow), and you’ve swapped out dbt-core execution for dbt Cloud execution (through the API).

    Familiarize your team with dbt Cloud's features and optimize development and deployment processes. Some key features to consider include:

    • Version management: Manage dbt versions and ensure team collaboration with dbt Cloud's one-click feature, removing the hassle of manual updates and version discrepancies. You can go Versionless to always get the latest features and early access to new functionality for your dbt project.
    • Development tools: Use the dbt Cloud CLI or dbt Cloud IDE to build, test, run, and version control your dbt projects.
    • Documentation and Source freshness: Automate storage of documentation and track source freshness in dbt Cloud, which streamlines project maintenance.
    • Notifications and logs: Receive immediate notifications for job failures, with direct links to the job details. Access comprehensive logs for all job runs to help with troubleshooting.
    • CI/CD: Use dbt Cloud's CI/CD feature to run your dbt projects in a temporary schema whenever new commits are pushed to open pull requests. This helps with catching bugs before deploying to production.

    Beyond your move

    Now that you’ve chosen dbt Cloud as your platform, you’ve unlocked the power of streamlining collaboration, enhancing workflow efficiency, and leveraging powerful features for analytics engineering teams. Here are some additional features you can use to unlock the full potential of dbt Cloud:

    • Audit logs: Use audit logs to review actions performed by people in your organization. Audit logs contain audited user and system events in real time. You can even export all the activity (beyond the 90 days you can view in dbt Cloud). enterprise
    • dbt Cloud APIs: Use dbt Cloud's robust APIs to create, read, update, and delete (CRUD) projects/jobs/environments project. The dbt Cloud Administrative API and Terraform provider facilitate programmatic access and configuration storage. While the Discovery API offers extensive metadata querying capabilities, such as job data, model configurations, usage, and overall project health. teamenterprise
    • dbt Explorer: Use dbt Explorer to view your project's resources (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. (Once you have a successful job in a Production environment). teamenterprise
    • dbt Semantic Layer: The dbt Semantic Layer allows you to define universal metrics on top of your models that can then be queried in your business intelligence (BI) tool. This means no more inconsistent metrics — there’s now a centralized way to define these metrics and create visibility in every component of the data flow. teamenterprise
    • dbt Mesh: Use dbt Mesh to share data models across organizations, enabling data teams to collaborate on shared data models and leverage the work of other teams. enterprise

    Additional help

    • dbt Learn courses: Access our free Learn dbt Cloud video courses for on-demand training.
    • dbt Community: Join the dbt Community to connect with other dbt users, ask questions, and share best practices.
    • dbt Support team: Our dbt Support team is always available to help you troubleshoot your dbt Cloud issues. Create a support ticket in dbt Cloud and we’ll be happy to help!
    • Account management Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. Book a demo to learn more. enterprise

    Summary

    This guide should now have given you some insight and equipped you with a framework for moving from dbt Core to dbt Cloud. This guide has covered the following key areas:

    • Considerations: Understanding the foundational steps required for a successful migration, including evaluating your current setup and identifying key considerations unique to your team's structure and workflow needs.

    • Plan you move: Highlighting the importance of workflow redesign, role-specific responsibilities, and the adoption of new processes to harness dbt Cloud's collaborative and efficient environment.

    • Move to dbt Cloud: Linking to the guide that outlines technical steps required to transition your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository.

    • Test and validate: Emphasizing technical transitions, including testing and validating your dbt projects within the dbt Cloud ecosystem to ensure data integrity and performance.

    • Transition and training: Share useful transition, training, and onboarding information for your team. Fully leverage dbt Cloud's capabilities, from development tools (dbt Cloud CLI and dbt Cloud IDE) to advanced features such as dbt Explorer, the Semantic Layer, and dbt Mesh.

    What’s next?

    Congratulations on finishing this guide, we hope it's given you insight into the considerations you need to take to best plan your move to dbt Cloud.

    For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:

    Guide
    Information
    Audience
    Move from dbt Core to dbt Cloud: What you need to knowUnderstand the considerations and methods needed in your move from dbt Core to dbt Cloud.Team leads
    Admins
    Move from dbt Core to dbt Cloud: Get startedLearn the steps needed to move from dbt Core to dbt Cloud.Developers
    Data engineers
    Data analysts
    Move from dbt Core to dbt Cloud: Optimization tipsLearn how to optimize your dbt Cloud experience with common scenarios and useful tips.Everyone

    Why move to dbt Cloud?

    If your team is using dbt Core today, you could be reading this guide because:

    • You’ve realized the burden of maintaining that deployment.
    • The person who set it up has since left.
    • You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices.

    Moving from dbt Core to dbt Cloud simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With dbt Cloud, you gain access to features like cross-team collaboration (dbt Mesh), version management, streamlined CI/CD, dbt Explorer for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently.

    It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.

    0