Create column processes

POST /api/meta/entity/bulk#createColumnProcesses

Create one or more column processes.

While processes are the basis for lineage in Atlan, column processes provide additional granular detail. When you visualize column-level lineage in Atlan, the column processes are the lines between columns that indicate the transformations that have occurred on input columns to produce output columns. Hence when defining a column process, you need to define the inputs to and outputs from the process.

The inputs and outputs specified need to already exist in Atlan before creating the column process.

Note that the creation relies on no matches being found for the provided qualifiedNames. If any of the qualifiedNames match any existing column process, those column processes will be updated rather than created. Also be aware that the qualifiedName is case-sensitive.

For more details on the semantics and other attributes available through this interface, see the Upsert entities operation.

Body Required

List of column processes to create.

  • entities array[object] Required

    Instance of a column process in Atlan, with its detailed information.

    • typeName string Required

      Should always be ColumnProcess.

    • attributes object Required

      Attributes that can exist in column process assets in Atlan.

      • qualifiedName string Required

        Unique name for this column process. This should be a concatenation of the parent process's qualifiedName, /column/ and the name of the column.

      • name string Required

        Human-readable name for this column process. This should be a concatenation of the fully-qualified name of the input column(s) with -> and the fully-qualified name of the output column(s).

      • connectionQualifiedName string | null Required

        Unique name of the parent connection through which this process exists.

      • code string | null

        Unused at column-level.

      • sql string | null

        Unused at column-level.

      • ast string | null

        Unused at column-level.

    • relationshipAttributes object | null Required

      Map of the relationships to this column process.

      • process object Required

        Details to use within an asset when referring to a process.

        • typeName string

          Should either be Process or a subtype of Process.

          Values are Process, ColumnProcess, or BIProcess.

        • guid string | null

          Unique identifier of the related process. If the uniqueAttributes are not provided, this must be provided.

        • uniqueAttributes object | null

          Attribute(s) that uniquely identify the related process. If the guid is not provided, these must be provided.

      • inputs array[object] | null Required

        Details to use within an asset when referring to a column.

        • typeName string

          Should always be Column.

        • guid string | null

          Unique identifier of the related column. If the uniqueAttributes are not provided, this must be provided.

        • uniqueAttributes object | null

          Attribute(s) that uniquely identify the related column. If the guid is not provided, these must be provided.

      • outputs array[object] | null Required

        Details to use within an asset when referring to a column.

        • typeName string

          Should always be Column.

        • guid string | null

          Unique identifier of the related column. If the uniqueAttributes are not provided, this must be provided.

        • uniqueAttributes object | null

          Attribute(s) that uniquely identify the related column. If the guid is not provided, these must be provided.

Responses

  • • 200 object

    Column processes that were successfully created or updated. Note from the example that even though we are creating a column process the response includes both updates to its inputs and outputs, and the creation of the column process itself.

    • mutatedEntities object | null

      Column processes that were changed.

      • CREATE array[object]

        Instance of a column process in Atlan, with its detailed information.

        • typeName string

          Should always be ColumnProcess.

        • attributes object | null

          Attributes that can exist in column process assets in Atlan.

          • qualifiedName string

            Unique name for this column process. This should be a concatenation of the parent process's qualifiedName, /column/ and the name of the column.

          • name string

            Human-readable name for this column process. This should be a concatenation of the fully-qualified name of the input column(s) with -> and the fully-qualified name of the output column(s).

          • displayName string | null

            Name used for display purposes (in user interfaces).

          • description string | null

            Description of the asset, as crawled from a source.

          • userDescription string | null

            Description of the asset, as provided by a user. If present, this will be used for the description in user interfaces. If not present, the description will be used.

          • tenantId string | null

            Name of the Atlan workspace in which the table exists.

          • certificateStatus string | null

            Status of the asset's certification.

            Values are VERIFIED, DRAFT, DEPRECATED, or null.

          • certificateStatusMessage string | null

            Human-readable descriptive message that can optionally be submitted when the certificateStatus is changed.

          • announcementTitle string | null

            Brief title for the announcement on this asset. Required when announcementType is specified.

          • announcementMessage string | null

            Detailed message to include in the announcement on this asset.

          • announcementType string | null

            Type of announcement on the asset.

            Values are information, warning, issue, or null.

          • ownerUsers array[string] | null

            List of users who own the asset.

          • ownerGroups array[string] | null

            List of groups who own the asset.

          • adminUsers array[string] | null

            List of users who administer the asset. (This is only used for Connection assets.)

          • adminGroups array[string] | null

            List of groups who administer the asset. (This is only used for Connection assets.)

          • viewerUsers array[string] | null
          • viewerGroups array[string] | null
          • connectorName string | null

            Name of the connector through which this asset is accessible.

          • connectionName string | null Deprecated

            Unused.

          • connectionQualifiedName string | null

            Unique name of the parent connection through which this process exists.

          • isDiscoverable boolean | null
          • isEditable boolean | null
          • subType object | null
          • viewScore number | null
          • popularityScore number | null
          • sourceOwners array[string] | null
          • sourceURL string | null
          • lastSyncWorkflowName string | null

            Name of the crawler that last synchronized this asset.

          • lastSyncRunAt integer | null

            Time (epoch) at which the table was last crawled, in milliseconds.

          • lastSyncRun string | null

            Name of the last run of the crawler that last synchronized this asset.

          • certificateUpdatedBy string | null

            Name of the user who last updated the certificateStatus.

          • certificateUpdatedAt integer | null

            Time (epoch) at which the certificateStatus was last updated, in milliseconds.

          • announcementUpdatedAt integer | null

            Time (epoch) at which the announcement was last updated, in milliseconds.

          • announcementUpdatedBy string | null

            User who last updated the announcement.

          • sourceCreatedBy string | null
          • sourceCreatedAt integer | null
          • sourceUpdatedAt integer | null
          • sourceUpdatedBy string | null
          • code string | null

            Unused at column-level.

          • sql string | null

            Unused at column-level.

          • ast string | null

            Unused at column-level.

        • classifications array[object] | null

          Instance of a classification in Atlan, with its detailed information.

          • typeName string

            Name of the classification. Note that this is the static-hashed unique name of the classification, not the human-readable displayName.

          • entityGuid string

            Unique identifier of the entity to which this classification is attached.

          • propagate boolean | null

            Whether to propagate this classification to other entities related to the entity to which the classification is attached.

          • removePropagationsOnEntityDelete boolean | null

            Whether to remove this classification from other entities to which it has been propagated when the classification is removed from this entity.

          • entityStatus string | null

            Status of the entity to which this classification is attached.

            Values are ACTIVE or DELETED.

        • displayText string | null

          Human-readable name of the entity.

        • guid string

          Unique identifier of the entity instance.

        • isIncomplete boolean | null

          Default value is false.

        • createdBy string | null

          Username of the user who created the object.

        • updatedBy string | null

          Username of the user who last updated the object.

        • createTime integer | null

          Time (epoch) at which this object was created, in milliseconds.

        • updateTime integer | null

          Time (epoch) at which this object was last updated, in milliseconds.

        • version integer | null

          Version of this object.

        • classificationNames array[string] | null

          List of classifications for this entity. Note that these are the internal hashed names used in Atlan, not the displayText of the classification.

        • labels array[string] | null

          Internal use only.

        • status string

          Status of the entity, either ACTIVE or DELETED.

          Values are ACTIVE or DELETED.

        • relationshipAttributes object | null

          Map of the relationships to this column process.

          • links array[object] | null

            Details to use within an asset when referring to a link (resource).

          • readme object

            Details to use within an asset when referring to a readme.

            • typeName string

              Should always be Readme.

            • guid string | null

              Unique identifier of the related readme. If the uniqueAttributes are not provided, this must be provided.

            • uniqueAttributes object | null

              Attribute(s) that uniquely identify the related readme. If the guid is not provided, these must be provided.

          • meanings array[object] | null

            Details to use within an asset when referring to a term.

            • typeName string

              Should always be AtlasnGlossaryTerm.

            • guid string | null

              Unique identifier of the related term. If the uniqueAttributes are not provided, this must be provided.

            • uniqueAttributes object | null

              Attribute(s) that uniquely identify the related term. If the guid is not provided, these must be provided.

              • qualifiedName string

                Unique name of the related term. Note that in Atlan this unique name is a hashed value, not the name you see in the UI.

          • inputs array[object] | null

            Details to use within an asset when referring to a column.

            • typeName string

              Should always be Column.

              Values are Readme, Link, Insight, Query, Database, Schema, Table, View, MaterialisedView, Column, Procedure, TablePartition, LookerDashboard, LookerExplore, LookerField, LookerFolder, LookerLook, LookerModel, LookerProject, LookerQuery, LookerTile, PowerBIDashboard, PowerBIDataflow, PowerBIDataset, PowerBIDatasource, PowerBIPage, PowerBIReport, PowerBITile, PowerBIWorkspace, SalesforceDashboard, SalesforceField, SalesforceObject, SalesforceOrganization, SalesforceReport, TableauCalculatedField, TableauDashboard, TableauDatasource, TableauDatasourceField, TableauFlow, TableauMetric, TableauProject, TableauSite, TableauWorkbook, TableauWorksheet, ObjectStore, S3, S3Bucket, S3Object, or DataStudioAsset.

            • guid string | null

              Unique identifier of the related column. If the uniqueAttributes are not provided, this must be provided.

            • uniqueAttributes object | null

              Attribute(s) that uniquely identify the related column. If the guid is not provided, these must be provided.

          • outputs array[object] | null

            Details to use within an asset when referring to a column.

            • typeName string

              Should always be Column.

              Values are Readme, Link, Insight, Query, Database, Schema, Table, View, MaterialisedView, Column, Procedure, TablePartition, LookerDashboard, LookerExplore, LookerField, LookerFolder, LookerLook, LookerModel, LookerProject, LookerQuery, LookerTile, PowerBIDashboard, PowerBIDataflow, PowerBIDataset, PowerBIDatasource, PowerBIPage, PowerBIReport, PowerBITile, PowerBIWorkspace, SalesforceDashboard, SalesforceField, SalesforceObject, SalesforceOrganization, SalesforceReport, TableauCalculatedField, TableauDashboard, TableauDatasource, TableauDatasourceField, TableauFlow, TableauMetric, TableauProject, TableauSite, TableauWorkbook, TableauWorksheet, ObjectStore, S3, S3Bucket, S3Object, or DataStudioAsset.

            • guid string | null

              Unique identifier of the related column. If the uniqueAttributes are not provided, this must be provided.

            • uniqueAttributes object | null

              Attribute(s) that uniquely identify the related column. If the guid is not provided, these must be provided.

          • columnProcesses array[object] | null

            Details to use within an asset when referring to a column process (used for column-level lineage).

            • typeName string

              Should always be ColumnProcess.

            • guid string | null

              Unique identifier of the related column process. If the uniqueAttributes are not provided, this must be provided.

            • uniqueAttributes object | null

              Attribute(s) that uniquely identify the related column process. If the guid is not provided, these must be provided.

              • qualifiedName string

                Unique name of the related column process. Note that in Atlan this unique name is a hashed value, not the name you see in the UI.

          • process object

            Details to use within an asset when referring to a process.

            • typeName string

              Should either be Process or a subtype of Process.

              Values are Process, ColumnProcess, or BIProcess.

            • guid string | null

              Unique identifier of the related process. If the uniqueAttributes are not provided, this must be provided.

            • uniqueAttributes object | null

              Attribute(s) that uniquely identify the related process. If the guid is not provided, these must be provided.

      • UPDATE array[object]
        Any of:
        Asset instance with detailed information array[object] Process instance with detailed response information array[object] Column process instance with detailed information array[object]
    • guidAssignments object

      Map of assigned unique identifiers for the changed column processes.

POST /api/meta/entity/bulk#createColumnProcesses 
curl \
 -X POST https://tenant.atlan.com/api/meta/entity/bulk#createColumnProcesses \
 -H "Authorization: Bearer $ACCESS_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"entities":[{"typeName":"ColumnProcess","attributes":{"name":"test_db/default/customer/id, test_db/default/sales/cst_id -\u003e test_db/default/sales_by_customer/customer_id","qualifiedName":"default/mongodb/mongodb.atlan.com/9745a6358bb500f03ab6f45142ee2c61/column/customer_id","connectorName":"mongodb","connectionName":"mongodb","connectionQualifiedName":"default/mongodb/mongodb.atlan.com"},"relationshipAttributes":{"inputs":[{"typeName":"Column","uniqueAttributes":{"qualifiedName":"default/mongodb/mongodb.atlan.com/test_db/default/customer/id"}},{"typeName":"Column","uniqueAttributes":{"qualifiedName":"default/mongodb/mongodb.atlan.com/test_db/default/sales/cst_id"}}],"outputs":[{"typeName":"Column","uniqueAttributes":{"qualifiedName":"default/mongodb/mongodb.atlan.com/test_db/default/sales_by_customer/customer_id"}}],"process":{"typeName":"Process","uniqueAttributes":{"qualifiedName":"default/mongodb/mongodb.atlan.com/9745a6358bb500f03ab6f45142ee2c61"}}}}]}'
Request example 
{
  "entities": [
    {
      "typeName": "ColumnProcess",
      "attributes": {
        "name": "test_db/default/customer/id, test_db/default/sales/cst_id -> test_db/default/sales_by_customer/customer_id",
        "qualifiedName": "default/mongodb/mongodb.atlan.com/9745a6358bb500f03ab6f45142ee2c61/column/customer_id",
        "connectorName": "mongodb",
        "connectionName": "mongodb",
        "connectionQualifiedName": "default/mongodb/mongodb.atlan.com"
      },
      "relationshipAttributes": {
        "inputs": [
          {
            "typeName": "Column",
            "uniqueAttributes": {
              "qualifiedName": "default/mongodb/mongodb.atlan.com/test_db/default/customer/id"
            }
          },
          {
            "typeName": "Column",
            "uniqueAttributes": {
              "qualifiedName": "default/mongodb/mongodb.atlan.com/test_db/default/sales/cst_id"
            }
          }
        ],
        "outputs": [
          {
            "typeName": "Column",
            "uniqueAttributes": {
              "qualifiedName": "default/mongodb/mongodb.atlan.com/test_db/default/sales_by_customer/customer_id"
            }
          }
        ],
        "process": {
          "typeName": "Process",
          "uniqueAttributes": {
            "qualifiedName": "default/mongodb/mongodb.atlan.com/9745a6358bb500f03ab6f45142ee2c61"
          }
        }
      }
    }
  ]
}
Response example (200) 
{
  "mutatedEntities": {
    "UPDATE": [
      {
        "typeName": "Column",
        "attributes": {
          "qualifiedName": "default/mongodb/mongodb.atlan.com/test_db/default/customer/id",
          "name": "id"
        },
        "guid": "4565d6bb-2d6f-4651-930b-74de9ae30f85",
        "status": "ACTIVE",
        "displayText": "id",
        "classificationNames": [],
        "meaningNames": [],
        "meanings": [],
        "isIncomplete": false,
        "labels": [],
        "createdBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "updatedBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "createTime": 1652796319437,
        "updateTime": 1652795647900
      },
      {
        "typeName": "Column",
        "attributes": {
          "qualifiedName": "default/mongodb/mongodb.atlan.com/test_db/default/sales/cst_id",
          "name": "cst_id"
        },
        "guid": "b9956cf9-4703-48bf-a6e0-b2772d566fa1",
        "status": "ACTIVE",
        "displayText": "cst_id",
        "classificationNames": [],
        "meaningNames": [],
        "meanings": [],
        "isIncomplete": false,
        "labels": [],
        "createdBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "updatedBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "createTime": 1652796319437,
        "updateTime": 1652795647900
      },
      {
        "typeName": "Column",
        "attributes": {
          "qualifiedName": "default/mongodb/mongodb.atlan.com/test_db/default/sales_by_customer/customer_id",
          "name": "customer_id"
        },
        "guid": "a61c0a63-1c40-4d4c-9d85-ded931f2205c",
        "status": "ACTIVE",
        "displayText": "customer_id",
        "classificationNames": [],
        "meaningNames": [],
        "meanings": [],
        "isIncomplete": false,
        "labels": [],
        "createdBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "updatedBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "createTime": 1652796319437,
        "updateTime": 1652795647900
      },
      {
        "typeName": "Process",
        "attributes": {
          "qualifiedName": "default/mongodb/mongodb.atlan.com/9745a6358bb500f03ab6f45142ee2c61",
          "name": "test_db/default/customer, test_db/default/sales -> test_db/default/sales_by_customer"
        },
        "guid": "dc14ffca-bf43-4731-b641-51408c4d892c",
        "status": "ACTIVE",
        "displayText": "test_db/default/customer, test_db/default/sales -> test_db/default/sales_by_customer",
        "classificationNames": [],
        "classifications": [],
        "meaningNames": [],
        "meanings": [],
        "isIncomplete": false,
        "labels": [],
        "createdBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "updatedBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "createTime": 1652795647900,
        "updateTime": 1652795647900
      }
    ],
    "CREATE": [
      {
        "typeName": "ColumnProcess",
        "attributes": {
          "popularityScore": 1.17549435e-38,
          "sourceCreatedAt": 0,
          "viewerUsers": [],
          "viewScore": 1.17549435e-38,
          "lastSyncRunAt": 0,
          "adminGroups": [],
          "qualifiedName": "default/mongodb/mongodb.atlan.com/9745a6358bb500f03ab6f45142ee2c61/column/customer_id",
          "__hasLineage": false,
          "adminUsers": [],
          "ownerGroups\"": [],
          "isEditable": true,
          "sourceUpdatedAt": 0,
          "announcementUpdatedAt": 0,
          "name": "test_db/default/customer/id, test_db/default/sales/cst_id -> test_db/default/sales_by_customer/customer_id",
          "certificateUpdatedAt": 0,
          "isDiscoverable": true,
          "viewerGroups": [],
          "ownerUsers": []
        },
        "guid": "ab5bccf8-67ff-4977-ad01-e9e449cb50f1",
        "status": "ACTIVE",
        "displayText": "test_db/default/customer/id, test_db/default/sales/cst_id -> test_db/default/sales_by_customer/customer_id",
        "classificationNames": [],
        "classifications": [],
        "meaningNames": [],
        "meanings": [],
        "isIncomplete": false,
        "labels": [],
        "createdBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "updatedBy": "service-account-apikey-bea1e1d1-e4e7-465e-8adf-7c61c227a59b",
        "createTime": 1652795647900,
        "updateTime": 1652795647900
      }
    ]
  },
  "guidAssignments": {
    "-10887525374266606": "4565d6bb-2d6f-4651-930b-74de9ae30f85",
    "-10887525374266607": "b9956cf9-4703-48bf-a6e0-b2772d566fa1",
    "-10887525374266608": "a61c0a63-1c40-4d4c-9d85-ded931f2205c",
    "-10887525374266609": "dc14ffca-bf43-4731-b641-51408c4d892c",
    "-10887525374266610": "ab5bccf8-67ff-4977-ad01-e9e449cb50f1"
  }
}