Error writing data to same destination in single data flow

I am getting the following error running a data flow that splits the input data into multiple streams and writes the results of each stream to the same destination table:

"This operation conflicts with another pending operation on this transaction. The operation failed."

The flow starts with a single source table with one row per student and multiple scores for that student. It does a few lookups and then splits the stream (using Multicast) in several layers, ultimately generating 25 destinations (one for each score to be recorded), all going to the same table (like a fact table). This all is running under a transaction at the package level, which is distributed to a separate machine.

Apparently, I cannot have all of these streams inserting data into the same table at one time. I don't understand why not. In an OLTP system, many transactions are inserting records into the same table at once. Why can't I do that within the same transaction?

I suppose I can use a UnionAll to join them back together before writing to a single destination, but that seems like an unnecessary waste and clutters the flow. Can anyone offer a different solution or a reason why this fails in the first place?

Thanks in advance.

What type of load are you using? Fast load with table lock? Certainly you will need to take off table lock, and maybe drop the fast load too.

An OLTP system is not doing a bulk load, it is doing nice and slow atomic transactions.

I also don't see what is wrong with the Union All transform. If you can get the current design working, then I think you should also try the union, and let us know which is faster, as that would be the real reason to choose one over the other, not good looks!

Re-reading this I see mention of a distributed transaction which matches the error better. Try using speparate connections for each destination. If the only reason for using the transaction is to ensure integrity accross the multiple destinations, then this would make me think of union even more as a distributed transaction can add a fair overhead.

|||

Thanks for the reply and the suggestions. Sorry for the delay in a follow-up, the automated alert for replies didn't work for me and I didn't know there was a reply.

I have since rewritten it to use the UnionAll task and it works. It's not so much an issue of how it looks, rather the trouble required to build it and maintain it. The current version of that task does not make it easy to distinguish which input is which so you can set the appropriate columns correctly. And if you should need to disconnect one stream from the UnionAll and reconnect it, the relative position of that stream moves and gets renamed. It's just a hassle to get right. Elegance in appearance is often times a clue to an elegant technical solution. Not always, but I'm a believer in a good "looking" solution.

You have a good suggestion about turning off the table lock and maybe the fast load option. I hadn't thought of that before I rewrote it. If I get a chance, I'll try the original approach and see if I can get it working.

The need for a transaction is to make sure all the data is loaded or none of it. If some part of it should fail, I want to rerun the entire package to load all the data again. This data flow isn't the only one in the package.

Thanks again.

Error writing data to same destination in single data flow

I am getting the following error running a data flow that splits the input data into multiple streams and writes the results of each stream to the same destination table:

"This operation conflicts with another pending operation on this transaction. The operation failed."

The flow starts with a single source table with one row per student and multiple scores for that student. It does a few lookups and then splits the stream (using Multicast) in several layers, ultimately generating 25 destinations (one for each score to be recorded), all going to the same table (like a fact table). This all is running under a transaction at the package level, which is distributed to a separate machine.

Apparently, I cannot have all of these streams inserting data into the same table at one time. I don't understand why not. In an OLTP system, many transactions are inserting records into the same table at once. Why can't I do that within the same transaction?

I suppose I can use a UnionAll to join them back together before writing to a single destination, but that seems like an unnecessary waste and clutters the flow. Can anyone offer a different solution or a reason why this fails in the first place?

Thanks in advance.

What type of load are you using? Fast load with table lock? Certainly you will need to take off table lock, and maybe drop the fast load too.

An OLTP system is not doing a bulk load, it is doing nice and slow atomic transactions.

I also don't see what is wrong with the Union All transform. If you can get the current design working, then I think you should also try the union, and let us know which is faster, as that would be the real reason to choose one over the other, not good looks!

Re-reading this I see mention of a distributed transaction which matches the error better. Try using speparate connections for each destination. If the only reason for using the transaction is to ensure integrity accross the multiple destinations, then this would make me think of union even more as a distributed transaction can add a fair overhead.

|||

Thanks for the reply and the suggestions. Sorry for the delay in a follow-up, the automated alert for replies didn't work for me and I didn't know there was a reply.

I have since rewritten it to use the UnionAll task and it works. It's not so much an issue of how it looks, rather the trouble required to build it and maintain it. The current version of that task does not make it easy to distinguish which input is which so you can set the appropriate columns correctly. And if you should need to disconnect one stream from the UnionAll and reconnect it, the relative position of that stream moves and gets renamed. It's just a hassle to get right. Elegance in appearance is often times a clue to an elegant technical solution. Not always, but I'm a believer in a good "looking" solution.

You have a good suggestion about turning off the table lock and maybe the fast load option. I hadn't thought of that before I rewrote it. If I get a chance, I'll try the original approach and see if I can get it working.

The need for a transaction is to make sure all the data is loaded or none of it. If some part of it should fail, I want to rerun the entire package to load all the data again. This data flow isn't the only one in the package.

Thanks again.

Error while converting Oracle Timestamp to Sql Server Timestamp (datetime) - "Invalid date

I am populating oracle source in Sql Server Destination. after few rows it fails it displays this error:

[OLE DB Destination [16]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server" Hresult: 0x80004005 Description:
"Invalid date format".

I used this script component using the following code in between the adapters, However after 9,500 rows it failed again giving the same above error:

To convert Oracle timestamp to Sql Server timestamp

If Row.CALCULATEDETADATECUST_IsNull = FalseThen

If IsDate(DateSerial(Row.CALCULATEDETADATECUST.Year, Row.CALCULATEDETADATECUST.Month, Row.CALCULATEDETADATECUST.Day)) Then

dt = Row.CALCULATEDETADATECUST

Row.CALCULATEDETADATECUSTD = dt

EndIf

EndIf

I don't know if my code is right . Please inform, how i can achieve this.

What is the value of the offending row? Redirect the error rows from the SQL dest and see what the offending value is.

The fact that it gets through 9500 rows and then dies means your logic is right but there is a dodgy row.|||

Thanks for your support, Crispin,

I redirected the output at destination (Fail Point) as you said and can see that some columns are displaying the Errors instead of the actual data that gives the clue about which is invalid)

Column 5

Error: Year, Month and day parameters describe an unrepresnatable date and time...

I doubt that the DateSerial function in the script component is unable to extract some formats..

So,

1) Does anybody know which representations and range of Oracle (timestamp) is not allowed in Sql Server (Datetime)
2) Do we have a stable code that can perform Oracle timestamp to Sql Server conversion here

Thanks

Subhash Subramanyam

|||

Hi Experts in SSIS forum,

Thanks to LoRez who has raised this question this month again.. I am still having the same problem in SSIS. I have an Oracle timestamp Source Columns that should be populated into Sql Server Datetime columns. Though I am aware of the ranges valid for both the cases, I was unable to frame it properly. . The expression (This only checks the Date Part , I wanted code that converts the time part of the Oracle timestamp into Sql Server Datetime) I have used to convert this is:

(!ISNULL(REQUESTED_ETA_DATE_CUST) && DATEPART("yyyy",REQUESTED_ETA_DATE_CUST) > 1752 && DATEPART("yyyy",REQUESTED_ETA_DATE_CUST) < 9999) ? REQUESTED_ETA_DATE_CUST : NULL(DT_DBTIMESTAMP)

Though the expression syntax is valid, it still gives error when I run the dataflow. The error is:

[Derived Column [136]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Derived Column" (136)" failed because error code 0xC0049067 occurred, and the error row disposition on "input column "REQUESTED_ETA_DATE_CUST" (372)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.

Though http://forums.informationbuilders.com/eve/forums/a/tpc/f/1381057331/m/7121008802 helps to create a trigger, but not the way we can achieve using SSIS.

Greatly Appreciate if anybody can pointer or paste the Expression or .NET code that can convert Oracle timestamp into Sql Server Datetime without issues

Thanks

Subhash Subramanyam

|||

Dear Umachandar, Thanks for briefing up the approach.

I'd appreciate if you go one step ahead to help my case work.Oracle Timestamp in my case is of the format 'MM/DD/YYYY hh:mm:ss.nnn' and I do not have any control over oracle source. Somehow I must be able to use this to populate the Sql Server datetime. We know that the year should be grater that 1752 which we can validate using above expression. But Validating TimePart seems to be difficult. I tried casting to string, but due to omission of leading zeros makes it difficult to substring that. So do you have a solution for this?

Umachandar's Reply

Oracle timestamp range subsumes that of SQL Server's datetime range. So you will not have any issues as long as you use the ISO 8601 format to specify the values (YYYY-MM-DDT hh:mm:ss.nnn). This will ensure that the value will be stored correctly irrespective of collation settings on either servers or Oracle session setttings. You can use timestamp with appropriate precision on Oracle side (timestamp(3) is closest) to match SQL Server datetime.

Migrating values from Oracle to SQL Server is a different ballgame. You will lose precision, values etc. Oracle has more richer support and wider ranges & ANSI SQL implementation.

Thanks

Subhash Subramanyam

|||

I've put up an article for this on my blog.

Cast this Oracle timestamp Column REQUESTED_ETA_DATE_CUST as below:

Decode(trunc((Extract(YEAR from REQUESTED_ETA_DATE_CUST))/1753), 0,'01/01/1753 12:00:00 AM', TO_CHAR(REQUESTED_ETA_DATE_CUST, 'MM/DD/YYYY hh:mms AM')) as REQUESTED_ETA_DATE_CUST

Or

Go for filtering valid ranges using SSIS transform after casting into a accepted datetime format.

Error while converting Oracle Timestamp to Sql Server Timestamp (datetime) - "Invalid date

I am populating oracle source in Sql Server Destination. after few rows it fails it displays this error:

[OLE DB Destination [16]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server" Hresult: 0x80004005 Description:
"Invalid date format".

I used this script component using the following code in between the adapters, However after 9,500 rows it failed again giving the same above error:

To convert Oracle timestamp to Sql Server timestamp

If Row.CALCULATEDETADATECUST_IsNull = False Then

If IsDate(DateSerial(Row.CALCULATEDETADATECUST.Year, Row.CALCULATEDETADATECUST.Month, Row.CALCULATEDETADATECUST.Day)) Then

dt = Row.CALCULATEDETADATECUST

Row.CALCULATEDETADATECUSTD = dt

End If

End If

I don't know if my code is right . Please inform, how i can achieve this.

What is the value of the offending row? Redirect the error rows from the SQL dest and see what the offending value is.

The fact that it gets through 9500 rows and then dies means your logic is right but there is a dodgy row.|||

Thanks for your support, Crispin,

I redirected the output at destination (Fail Point) as you said and can see that some columns are displaying the Errors instead of the actual data that gives the clue about which is invalid)

Column 5

Error: Year, Month and day parameters describe an unrepresnatable date and time...

I doubt that the DateSerial function in the script component is unable to extract some formats..

So,

1) Does anybody know which representations and range of Oracle (timestamp) is not allowed in Sql Server (Datetime)
2) Do we have a stable code that can perform Oracle timestamp to Sql Server conversion here

Thanks

Subhash Subramanyam

|||

Hi Experts in SSIS forum,

Thanks to LoRez who has raised this question this month again.. I am still having the same problem in SSIS. I have an Oracle timestamp Source Columns that should be populated into Sql Server Datetime columns. Though I am aware of the ranges valid for both the cases, I was unable to frame it properly. . The expression (This only checks the Date Part , I wanted code that converts the time part of the Oracle timestamp into Sql Server Datetime) I have used to convert this is:

(!ISNULL(REQUESTED_ETA_DATE_CUST) && DATEPART("yyyy",REQUESTED_ETA_DATE_CUST) > 1752 && DATEPART("yyyy",REQUESTED_ETA_DATE_CUST) < 9999) ? REQUESTED_ETA_DATE_CUST : NULL(DT_DBTIMESTAMP)

Though the expression syntax is valid, it still gives error when I run the dataflow. The error is:

[Derived Column [136]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Derived Column" (136)" failed because error code 0xC0049067 occurred, and the error row disposition on "input column "REQUESTED_ETA_DATE_CUST" (372)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.

Though http://forums.informationbuilders.com/eve/forums/a/tpc/f/1381057331/m/7121008802 helps to create a trigger, but not the way we can achieve using SSIS.

Greatly Appreciate if anybody can pointer or paste the Expression or .NET code that can convert Oracle timestamp into Sql Server Datetime without issues

Thanks

Subhash Subramanyam

|||

Dear Umachandar, Thanks for briefing up the approach.

I'd appreciate if you go one step ahead to help my case work.Oracle Timestamp in my case is of the format 'MM/DD/YYYY hh:mm:ss.nnn' and I do not have any control over oracle source. Somehow I must be able to use this to populate the Sql Server datetime. We know that the year should be grater that 1752 which we can validate using above expression. But Validating TimePart seems to be difficult. I tried casting to string, but due to omission of leading zeros makes it difficult to substring that. So do you have a solution for this?

Umachandar's Reply

Oracle timestamp range subsumes that of SQL Server's datetime range. So you will not have any issues as long as you use the ISO 8601 format to specify the values (YYYY-MM-DDT hh:mm:ss.nnn). This will ensure that the value will be stored correctly irrespective of collation settings on either servers or Oracle session setttings. You can use timestamp with appropriate precision on Oracle side (timestamp(3) is closest) to match SQL Server datetime.

Migrating values from Oracle to SQL Server is a different ballgame. You will lose precision, values etc. Oracle has more richer support and wider ranges & ANSI SQL implementation.

Thanks

Subhash Subramanyam

|||

I've put up an article for this on my blog.

Cast this Oracle timestamp Column REQUESTED_ETA_DATE_CUST as below:

Decode(trunc((Extract(YEAR from REQUESTED_ETA_DATE_CUST))/1753), 0,'01/01/1753 12:00:00 AM', TO_CHAR(REQUESTED_ETA_DATE_CUST, 'MM/DD/YYYY hh:mms AM')) as REQUESTED_ETA_DATE_CUST

Or

Go for filtering valid ranges using SSIS transform after casting into a accepted datetime format.

Error while converting Oracle Timestamp to Sql Server Timestamp (datetime) - "Invalid date

I am populating oracle source in Sql Server Destination. after few rows it fails it displays this error:

[OLE DB Destination [16]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server" Hresult: 0x80004005 Description:
"Invalid date format".

I used this script component using the following code in between the adapters, However after 9,500 rows it failed again giving the same above error:

To convert Oracle timestamp to Sql Server timestamp

If Row.CALCULATEDETADATECUST_IsNull = False Then

If IsDate(DateSerial(Row.CALCULATEDETADATECUST.Year, Row.CALCULATEDETADATECUST.Month, Row.CALCULATEDETADATECUST.Day)) Then

dt = Row.CALCULATEDETADATECUST

Row.CALCULATEDETADATECUSTD = dt

End If

End If

I don't know if my code is right . Please inform, how i can achieve this.

What is the value of the offending row? Redirect the error rows from the SQL dest and see what the offending value is.

The fact that it gets through 9500 rows and then dies means your logic is right but there is a dodgy row.|||

Thanks for your support, Crispin,

I redirected the output at destination (Fail Point) as you said and can see that some columns are displaying the Errors instead of the actual data that gives the clue about which is invalid)

Column 5

Error: Year, Month and day parameters describe an unrepresnatable date and time...

I doubt that the DateSerial function in the script component is unable to extract some formats..

So,

1) Does anybody know which representations and range of Oracle (timestamp) is not allowed in Sql Server (Datetime)
2) Do we have a stable code that can perform Oracle timestamp to Sql Server conversion here

Thanks

Subhash Subramanyam

|||

Hi Experts in SSIS forum,

Thanks to LoRez who has raised this question this month again.. I am still having the same problem in SSIS. I have an Oracle timestamp Source Columns that should be populated into Sql Server Datetime columns. Though I am aware of the ranges valid for both the cases, I was unable to frame it properly. . The expression (This only checks the Date Part , I wanted code that converts the time part of the Oracle timestamp into Sql Server Datetime) I have used to convert this is:

(!ISNULL(REQUESTED_ETA_DATE_CUST) && DATEPART("yyyy",REQUESTED_ETA_DATE_CUST) > 1752 && DATEPART("yyyy",REQUESTED_ETA_DATE_CUST) < 9999) ? REQUESTED_ETA_DATE_CUST : NULL(DT_DBTIMESTAMP)

Though the expression syntax is valid, it still gives error when I run the dataflow. The error is:

[Derived Column [136]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Derived Column" (136)" failed because error code 0xC0049067 occurred, and the error row disposition on "input column "REQUESTED_ETA_DATE_CUST" (372)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.

Though http://forums.informationbuilders.com/eve/forums/a/tpc/f/1381057331/m/7121008802 helps to create a trigger, but not the way we can achieve using SSIS.

Greatly Appreciate if anybody can pointer or paste the Expression or .NET code that can convert Oracle timestamp into Sql Server Datetime without issues

Thanks

Subhash Subramanyam

|||

Dear Umachandar, Thanks for briefing up the approach.

I'd appreciate if you go one step ahead to help my case work.Oracle Timestamp in my case is of the format 'MM/DD/YYYY hh:mm:ss.nnn' and I do not have any control over oracle source. Somehow I must be able to use this to populate the Sql Server datetime. We know that the year should be grater that 1752 which we can validate using above expression. But Validating TimePart seems to be difficult. I tried casting to string, but due to omission of leading zeros makes it difficult to substring that. So do you have a solution for this?

Umachandar's Reply

Oracle timestamp range subsumes that of SQL Server's datetime range. So you will not have any issues as long as you use the ISO 8601 format to specify the values (YYYY-MM-DDT hh:mm:ss.nnn). This will ensure that the value will be stored correctly irrespective of collation settings on either servers or Oracle session setttings. You can use timestamp with appropriate precision on Oracle side (timestamp(3) is closest) to match SQL Server datetime.

Migrating values from Oracle to SQL Server is a different ballgame. You will lose precision, values etc. Oracle has more richer support and wider ranges & ANSI SQL implementation.

Thanks

Subhash Subramanyam

|||

I've put up an article for this on my blog.

Cast this Oracle timestamp Column REQUESTED_ETA_DATE_CUST as below:

Decode(trunc((Extract(YEAR from REQUESTED_ETA_DATE_CUST))/1753), 0,'01/01/1753 12:00:00 AM', TO_CHAR(REQUESTED_ETA_DATE_CUST, 'MM/DD/YYYY hh:mms AM')) as REQUESTED_ETA_DATE_CUST

Or

Go for filtering valid ranges using SSIS transform after casting into a accepted datetime format.