Semantic discovery and mapping between data sources

{
  "publication_number": "US9336253B2",
  "country": "US",
  "kind": "B2",
  "title": "Semantic discovery and mapping between data sources",
  "abstract": "An apparatus and method are described for the discovery of semantics, relationships and mappings between data in different software applications, databases, files, reports, messages, or systems. In one aspect, semantics and relationships and mappings are identified between a first and a second data source. A binding condition is discovered between portions of data in the first and the second data source. The binding condition is used to discover correlations between portions of data in the first and the second data source. The binding condition and the correlations are used to discover a transformation function between portions of data in the first and the second data source.",
  "claims": [
    "1. A method for determining a functional correlation between columns of a first data source and a second data source, the method comprising: combining the first data source and the second data source into a combined data source based on a binding condition; identifying a correlation count for columns of the first data source and the second data source in the combined data source; comparing the correlation count to a threshold value; and determining the functional correlation between columns of the first data source and the second data source, when the correlation count is more than the threshold value.",
    "2. The method of claim 1, wherein the binding condition comprises an expression on attributes of data objects in the first and second data sources that identifies which instances of the data objects in the first data source map to instances of the data objects in the second data source.",
    "3. The method of claim 1, wherein the combining the first and second data source based on a binding condition comprises an outer join between the first and the second data sources.",
    "4. The method of claim 1, wherein the correlation count identifies different data objects in the first and second data sources that correspond to each other in the first and second data sources.",
    "5. The method of claim 1, wherein the functional correlation is defined by a stateless transformation function that transforms data objects in the first data source to different data objects in the second data source.",
    "6. The method of claim 1, further comprising: determining correlation counts for all binding conditions; determining whether a highest correlation count of the binding conditions is greater than the threshold value; and indicating that there is no binding between the first and second data sources in response to determining that the highest correlation count is less than the threshold value.",
    "7. The method of claim 6, wherein the binding condition having the highest correlation count is selected as a primary binding condition to use to determine the functional correlation in response determining that the highest correlation count is greater than the threshold value.",
    "8. A system in communication with a first data source and a second data source, comprising: a processor; and a computer storage device having program instructions executed by the processor to determine a functional correlation between columns of the first data source and the second data source by perform operations comprising: combining the first data source and the second data source into a combined data source based on a binding condition; identifying a correlation count for columns of the first data source and the second data source in the combined data source; comparing the correlation count to a threshold value; and determining the functional correlation between columns of the first data source and the second data source, when the correlation count is more than the threshold value.",
    "9. The system of claim 8, wherein the binding condition comprises an expression on attributes of data objects in the first and second data sources that identifies which instances of the data objects in the first data source map to instances of the data objects in the second data source.",
    "10. The system of claim 8, wherein the combining the first and second data source based on a binding condition comprises an outer join between the first and the second data sources.",
    "11. The system of claim 8, wherein the correlation count identifies different data objects in the first and second data sources that correspond to each other in the first and second data sources.",
    "12. The system of claim 8, wherein the functional correlation is defined by a stateless transformation function that transforms data objects in the first data source to different data objects in the second data source.",
    "13. The system of claim 8, wherein the operations further comprise: determining correlation counts for all binding conditions; determining whether a highest correlation count of the binding conditions is greater than the threshold value; and indicating that there is no binding between the first and second data sources in response to determining that the highest correlation count is less than the threshold value.",
    "14. The system of claim 13, wherein the binding condition having the highest correlation count is selected as a primary binding condition to use to determine the functional correlation in response determining that the highest correlation count is greater than the threshold value.",
    "15. A computer readable storage device comprising executable program instructions executed by a processor to determine a functional correlation between columns of a first data source and a second data source by performing operations, the operations comprising: combining the first data source and the second data source into a combined data source based on a binding condition; identifying a correlation count for columns of the first data source and the second data source in the combined data source; comparing the correlation count to a threshold value; and determining the functional correlation between columns of the first data source and the second data source, when the correlation count is more than the threshold value.",
    "16. The computer readable storage device of claim 15, wherein the binding condition comprises an expression on attributes of data objects in the first and second data sources that identifies which instances of the data objects in the first data source map to instances of the data objects in the second data source.",
    "17. The computer readable storage device of claim 15, wherein the combining the first and second data sources based on a binding condition comprises an outer join between the first and the second data sources.",
    "18. The computer readable storage device of claim 15, wherein the correlation count identifies different data objects in the first and second data sources that correspond to each other in the first and second data sources.",
    "19. The computer readable storage device of claim 15, wherein the functional correlation is defined by a stateless transformation function that transforms data objects in the first data source to different data objects in the second data source.",
    "20. The computer readable storage device of claim 15, wherein the operations further comprise: determining correlation counts for all binding conditions; determining whether a highest correlation count of the binding conditions is greater than the threshold value; and indicating that there is no binding between the first and second data sources in response to determining that the highest correlation count is less than the threshold value.",
    "21. The computer readable storage device of claim 20, wherein the binding condition having the highest correlation count is selected as a primary binding condition to use to determine the functional correlation in response determining that the highest correlation count is greater than the threshold value."
  ],
  "description_excerpt": "The present invention relates to a method and apparatus for automating the way computer systems, applications, files and databases are integrated. Specifically, the present invention relates to the discovery of semantics, relationships and mappings between data in different software applications, databases, files, reports, messages or systems.\n\nThe Information Technology (IT) professionals have performed data and application integration for many years. A typical integration project has three distinct phases: discovery, integration, and maintenance. Discovery phase involves identifying relationships between the systems that need to be integrated. Integration phase involves creating programs or specifications to perform the physical data movement or interfacing. The maintenance phase involves updating and changing the integration programs to correspond to changes in the systems being integrated or to accommodate new integration requirements.\n\nSeveral prior art patents describe various conventional ways of integrating data across systems. U.S. Pat. No. 5,675,785 - Hall, et al., Oct. 10, 1997, 395/613, “DATA WAREHOUSE WHICH IS ACCESSED BY A USER USING A SCHEMA OF VERTICAL TABLES”. This patent describes a system where a layer of logical tables is created and mapped to the physical tables in a data warehouse such that the user specifies queries against the logical tables to access data in the physical tables. It does not address the problem of discovering relationships and mappings between data in different data sources.",
  "cpc": [
    "G06F 16/221",
    "G06F 16/00",
    "G06F 16/211",
    "G06F 16/2237",
    "G06F 16/24544",
    "G06F 16/2468",
    "G06F 17/30",
    "G06F 17/30292",
    "G06F 17/30315",
    "G06F 17/30324",
    "G06F 17/30466",
    "G06F 17/30542",
    "Y10S 707/99942",
    "Y10S 707/99943",
    "Y10S 707/99944",
    "Y10S 707/99945"
  ],
  "ipc": [
    "G06F 17/30"
  ],
  "assignees": [
    "International Business Machines Corp"
  ],
  "inventors": [
    "Alexander Gorelik",
    "Lingling Yan"
  ],
  "filing_date": "2014-10-06",
  "publication_date": "2016-05-10",
  "grant_date": "2016-05-10",
  "priority_date": "2003-09-10",
  "application_number": "US-201414507805-A",
  "family_id": "34228902",
  "cited_by_count": 53,
  "citations": [
    "US5594900A",
    "US5809297A",
    "US5692107A",
    "US5675785A",
    "US5794209A",
    "US5615341A",
    "US6026392A",
    "US5806066A",
    "US5813002A",
    "US5943667A",
    "US6226649B1",
    "US6272478B1",
    "US5978796A",
    "US6112198A",
    "US6092064A",
    "US6339775B1",
    "US6301575B1",
    "US6311173B1",
    "US6298342B1",
    "US6594653B2",
    "US6078918A",
    "US6049797A",
    "US6185549B1",
    "US6317735B1",
    "US6182070B1",
    "US6311179B1",
    "US20060253435A1",
    "US6278997B1",
    "US7065517B1",
    "US6609123B1",
    "US6965888B1",
    "US6604095B1",
    "WO2001037135A2",
    "US6393424B1",
    "US7007020B1",
    "WO2001075679A1",
    "US20020091707A1",
    "US6542881B1",
    "US6850947B1",
    "US6954756B2",
    "US6941303B2",
    "US6836773B2",
    "US6877012B2",
    "WO2002073468A1",
    "US20020178170A1",
    "US20040093344A1",
    "US20040093559A1",
    "US20030126056A1",
    "US20030115280A1",
    "US20030158850A1",
    "US20030191667A1",
    "US20030212678A1",
    "US6973459B1",
    "US20030217069A1",
    "US7249118B2",
    "US7028288B2",
    "US20040015783A1",
    "US20050066240A1",
    "US7644361B2",
    "CN1145901C",
    "CN1435781A",
    "US20040189708A1",
    "US20040226002A1",
    "US7680828B2",
    "US20060271528A1",
    "US20090094274A1",
    "US7426520B2",
    "US20120158745A1",
    "US20050055369A1",
    "US20130254183A1",
    "US20050060313A1",
    "US20050066263A1",
    "US20050108631A1",
    "US20050086250A1",
    "US20050144552A1",
    "US7266537B2",
    "US20050182739A1",
    "US20050256892A1",
    "US20050234688A1",
    "US20060106847A1",
    "US20050278308A1",
    "US20060004740A1",
    "US20060004750A1",
    "US7490106B2",
    "US20060053382A1",
    "US20060136462A1",
    "US20060136461A1",
    "US20060161546A1",
    "US20060167579A1",
    "US20060274760A1",
    "US20080027899A1",
    "US20070073688A1",
    "US20070106785A1",
    "US20070156790A1",
    "US20070179959A1",
    "US20070239769A1",
    "US20070294221A1",
    "US20080005183A1",
    "US20080085742A1",
    "US7836004B2",
    "US20080243823A1",
    "US8166000B2",
    "US8171001B2",
    "US20090024551A1",
    "US7702694B1",
    "US20090204631A1",
    "US20090281974A1",
    "US20090327208A1",
    "US20110246503A1",
    "US20120078913A1"
  ]
}