Details

    • Type: Task Task
    • Status: Waiting for Infra
    • Priority: Major Major
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: JIRA
    • Labels:
      None

      Description

      Files representing the Adobe Flex JIRA database has been made available to Tony Stevenson and infrastructure@a.o via DropBox.

      The FLEX.zip is the 6GB zip of the attachments. According to an MD5 app I got off the internet, the checksum is: 12c4b40f0ada7aa7145e52d338399441
      The FlexSDKENTDMV.zip is the import file. It's checksum is: a8118ac7e2feb452eb6281091ec09582

      Please perform a Project Import of these files into the FLEX project at issues.a.o.
      1. catalina.out
        9.34 MB
        Tony Stevenson
      2. flex-20120201.xml
        31 kB
        Bertrand Delacretaz

        Activity

        Hide
        Alex Harui added a comment -
        Still holding, waiting for me to find the time to build the utility that will rename the attachments. If you want to close it out you can and I'll open a new JIRA when I'm ready.

        Thanks,
        -Alex
        Show
        Alex Harui added a comment - Still holding, waiting for me to find the time to build the utility that will rename the attachments. If you want to close it out you can and I'll open a new JIRA when I'm ready. Thanks, -Alex
        Hide
        Tony Stevenson added a comment -
        Ok,

        So 2 months on, I just want to check if anything can be updated here? Or are we will in a holding pattern?
        Show
        Tony Stevenson added a comment - Ok, So 2 months on, I just want to check if anything can be updated here? Or are we will in a holding pattern?
        Hide
        Alex Harui added a comment -
        Hi Tony,

        Right now, every couple of months, someone wants an old attachment. I go find it in my local JIRA copy and attach it manually and we continue on. That seems manageable for now, but if more committers showed up and asked more often then that would motivate me to finally do something about it.

        Now if you want to just import everything under a new KEY, I think that would be ok, but I would worry about potential confusion if those JIRA issues get into search results.

        So, I don't think we can forget the whole thing, but for me, it just hasn't gotten priority right now. If you're ok just letting this sit for now, I'm ok with that too.
        Show
        Alex Harui added a comment - Hi Tony, Right now, every couple of months, someone wants an old attachment. I go find it in my local JIRA copy and attach it manually and we continue on. That seems manageable for now, but if more committers showed up and asked more often then that would motivate me to finally do something about it. Now if you want to just import everything under a new KEY, I think that would be ok, but I would worry about potential confusion if those JIRA issues get into search results. So, I don't think we can forget the whole thing, but for me, it just hasn't gotten priority right now. If you're ok just letting this sit for now, I'm ok with that too.
        Hide
        Tony Stevenson added a comment -
        Alex,

        Are we pissing into the wind with this one now? In other words are we going to gain anything still despite the significant pain levels on both sides?

        Do we want to continue? Do we just want to get an export in a new KEY name, and leave them side by side? Or just forget the whole thing?
        Show
        Tony Stevenson added a comment - Alex, Are we pissing into the wind with this one now? In other words are we going to gain anything still despite the significant pain levels on both sides? Do we want to continue? Do we just want to get an export in a new KEY name, and leave them side by side? Or just forget the whole thing?
        Hide
        Alex Harui added a comment -
        IIRC, the plan was for me to give you a zip of attachments that have been renamed to real names, and you would post it to some subdomain of a.o. I was going to do this by parsing the XML file to figure out the names. I have had zero cycles to work on it.

        If you/Humbedooh have the cycles, you are welcome to take it on.

        You're also welcome to try to add the attachments some other way, but well over 200 bugs have been modified since the original import and I wouldn't want to lose those changes.

        Thanks,
        -Alex
        Show
        Alex Harui added a comment - IIRC, the plan was for me to give you a zip of attachments that have been renamed to real names, and you would post it to some subdomain of a.o. I was going to do this by parsing the XML file to figure out the names. I have had zero cycles to work on it. If you/Humbedooh have the cycles, you are welcome to take it on. You're also welcome to try to add the attachments some other way, but well over 200 bugs have been modified since the original import and I wouldn't want to lose those changes. Thanks, -Alex
        Hide
        ASF IRC Bot added a comment -
        Comment from pctony via IRC:
        Alex, do you still want to try this? If so, we can talk about doing this for you - I can ask Humbedooh to poke it for you as he is now (lucky him) getting more exposure to the jira import/export pain cycle. :)
        Show
        ASF IRC Bot added a comment - Comment from pctony via IRC: Alex, do you still want to try this? If so, we can talk about doing this for you - I can ask Humbedooh to poke it for you as he is now (lucky him) getting more exposure to the jira import/export pain cycle. :)
        Hide
        Gavin added a comment -
        [~pctony] ping
        Show
        Gavin added a comment - [~pctony] ping
        Hide
        Alex Harui added a comment -
        OK, that doesn't sound too practical. Is the SOAP-RPC available on JIRA? What would happen if I tried to suck all 34000 issues onto my computer via SOAP? Only the last 400 might have attachments.

        Otherwise, I guess I will go back to the community and see if we can just import the old issues and attachments into a new project.
        Show
        Alex Harui added a comment - OK, that doesn't sound too practical. Is the SOAP-RPC available on JIRA? What would happen if I tried to suck all 34000 issues onto my computer via SOAP? Only the last 400 might have attachments. Otherwise, I guess I will go back to the community and see if we can just import the old issues and attachments into a new project.
        Hide
        Tony Stevenson added a comment -
        Alex,

        It is doable, as for practical that is another matter. The sheer scale of the export and then import is at least 18 hours of cpu time (almost entirely import). This was on the production VM a while back, the smaller test instance VMs will need scaling up to support the disk space and ram to cope with it.

        We cannot shutdown a single instance of a project. Aside from asking people nicely not to touch it. If we need to take JIRA offline for a long period we need schedule this outwards.

        How do you propose to get the files to match the XML? Im not sure that you can track them back through?
        Show
        Tony Stevenson added a comment - Alex, It is doable, as for practical that is another matter. The sheer scale of the export and then import is at least 18 hours of cpu time (almost entirely import). This was on the production VM a while back, the smaller test instance VMs will need scaling up to support the disk space and ram to cope with it. We cannot shutdown a single instance of a project. Aside from asking people nicely not to touch it. If we need to take JIRA offline for a long period we need schedule this outwards. How do you propose to get the files to match the XML? Im not sure that you can track them back through?
        Hide
        Alex Harui added a comment -
        Hi Tony,

        I thought you could do the full export (which I'm sure is monumental) then import just FLEX into a test instance from that monumental export. Then when you export that test instance it will have just what I need. My plan is to take the files I have and use the export file from that test instance and to make one new import file. I also need a zip of the FLEX attachments folder because I think I have to rename attachments and create a new attachments zip file that combines the ones I have with the ones already in Apache Flex JIRA.

        Is that a practical thing for you to do? Keep in mind, we need to do this at least twice: once now so I can write a tool that will generate the right xml and attachments zip, and again when I have the tool working since we will want to shutdown Apache Flex JIRA long enough for you to give me the latest xml and attachments zip, generate the new one, have you import it, and let me test it. I don't want to shut down Apache Flex JIRA right now since it will probably take me several days to write and test this tool and new bugs are being filed daily.

        Thanks,
        -Alex
        Show
        Alex Harui added a comment - Hi Tony, I thought you could do the full export (which I'm sure is monumental) then import just FLEX into a test instance from that monumental export. Then when you export that test instance it will have just what I need. My plan is to take the files I have and use the export file from that test instance and to make one new import file. I also need a zip of the FLEX attachments folder because I think I have to rename attachments and create a new attachments zip file that combines the ones I have with the ones already in Apache Flex JIRA. Is that a practical thing for you to do? Keep in mind, we need to do this at least twice: once now so I can write a tool that will generate the right xml and attachments zip, and again when I have the tool working since we will want to shutdown Apache Flex JIRA long enough for you to give me the latest xml and attachments zip, generate the new one, have you import it, and let me test it. I don't want to shut down Apache Flex JIRA right now since it will probably take me several days to write and test this tool and new bugs are being filed daily. Thanks, -Alex
        Hide
        Tony Stevenson added a comment -
        Alex,

        Just to let you know I have not forgotten about this. I have just been trying to find a way to do it, and to be honest I'm not exactly coming up with much.
        I can run off an export, but it is truly monumental, plus I'd somehow have to strip out literally 10's of thousands of entities to clean it of personal data etc.

        Do you have any ideas?
        Show
        Tony Stevenson added a comment - Alex, Just to let you know I have not forgotten about this. I have just been trying to find a way to do it, and to be honest I'm not exactly coming up with much. I can run off an export, but it is truly monumental, plus I'd somehow have to strip out literally 10's of thousands of entities to clean it of personal data etc. Do you have any ideas?
        Hide
        Alex Harui added a comment -
        FlexAttachments.tar.gz is now available in a private URL. The MD5 is: 97d27d035e7dadcd75f04525d8643e4a
        Show
        Alex Harui added a comment - FlexAttachments.tar.gz is now available in a private URL. The MD5 is: 97d27d035e7dadcd75f04525d8643e4a
        Hide
        Tony Stevenson added a comment - - edited
        {QUOTE}

        Alex,

        So I took your zip file onto a mac exploded it using the UI, and using unzip on the terminal.
        They produced exactly the same output.

        I then did a clean install of JIRA 5.0.4, and started the project import. See the attached catalina log file. This is from a clean start, and attempted import.

        root@arcas [/x1/flex-jira-dance/jira/logs ] cat catalina.out| grep 'FLEX/FLEX-' | wc -l
        28267

        That shows we have 28267 missing files. Something isn't right here.

        root@arcas [/x1/flex-jira-dance/jira/logs ] cat catalina.out| grep 'FLEX/FLEX-' | head -n3

        2013-02-04 10:32:23,283 JiraTaskExectionThread-2 WARN pctony 632x30x1 pd7roa 128.232.134.143 /secure/admin/ProjectImportSelectProject.jspa [imports.project.handler.AttachmentFileValidatorHandler] The attachment 'rotateBug.zip' does not exist at '/x1/flex-jira-dance/data/import/attachments/FLEX/FLEX-75/10000'. It will not be imported.

        2013-02-04 10:32:23,284 JiraTaskExectionThread-2 WARN pctony 632x30x1 pd7roa 128.232.134.143 /secure/admin/ProjectImportSelectProject.jspa [imports.project.handler.AttachmentFileValidatorHandler] The attachment 'workaround.mxml' does not exist at '/x1/flex-jira-dance/data/import/attachments/FLEX/FLEX-305/10001'. It will not be imported.

        2013-02-04 10:32:23,284 JiraTaskExectionThread-2 WARN pctony 632x30x1 pd7roa 128.232.134.143 /secure/admin/ProjectImportSelectProject.jspa [imports.project.handler.AttachmentFileValidatorHandler] The attachment 'BusyCursorTest.zip' does not exist at '/x1/flex-jira-dance/data/import/attachments/FLEX/FLEX-313/10002'. It will not be imported.

        That is 3 missing files at the start of the import.

        root@arcas [/x1/flex-jira-dance/jira/logs ] ls -lah /x1/flex-jira-dance/data/import/attachments/FLEX/FLEX-75/
        total 628K
        drwxr-xr-x 2 root root 4.0K Feb 4 09:52 .
        drwxr-xr-x 12686 root root 356K Feb 4 09:56 ..
        -rwxr-xr-x 1 root root 261K Feb 4 09:52 10186

        root@arcas [/x1/flex-jira-dance/jira/logs ] ls -lah /x1/flex-jira-dance/data/import/attachments/FLEX/FLEX-305
        total 368K
        drwxr-xr-x 2 root root 4.0K Feb 4 09:51 .
        drwxr-xr-x 12686 root root 356K Feb 4 09:56 ..
        -rwxr-xr-x 1 root root 1.8K Feb 4 09:51 10187

        root@arcas [/x1/flex-jira-dance/jira/logs ] ls -lah /x1/flex-jira-dance/data/import/attachments/FLEX/FLEX-313
        total 380K
        drwxr-xr-x 2 root root 4.0K Feb 4 09:53 .
        drwxr-xr-x 12686 root root 356K Feb 4 09:56 ..
        -rwxr-xr-x 1 root root 14K Feb 4 09:53 10188


        As you can see none of the blob objects in the zip file match. None. Did you do some kind of transforms on this data?
        The file that the import is looking for is taken from the xml import file (in the zip). Did you import the data into another version of jira to export only the FLEX project, i.e. filter out others. If so you have to import the attachments into that instance as they will get renumbered starting at 10000 - if not the attachments we have are from the original and mismatch your exported zip/xml file.

        As it stands I cannot use the attachments.

        The attachments on disk do not use files names as per the source file. They are stored as blobs on disk starting at 10000, and the filename is stored in SQL.


        Alex, as far as I can tell. You did the export from Adobe, into a staging instance and then exported again after some filtering I suspect. That will only work if you import the the files as part of your import/export process.

        So far I have spent over 75 hours on this since the start of the year. Is there anyway we can start the process again? You get a clean dump and attachments? If so please can we get it as a tar.gz ?

        I hope this all makes sense. If not just shout.

        {QUOTE}
        Show
        Tony Stevenson added a comment - - edited {QUOTE} Alex, So I took your zip file onto a mac exploded it using the UI, and using unzip on the terminal. They produced exactly the same output. I then did a clean install of JIRA 5.0.4, and started the project import. See the attached catalina log file. This is from a clean start, and attempted import. root@arcas [/x1/flex-jira-dance/jira/logs ] cat catalina.out| grep 'FLEX/FLEX-' | wc -l 28267 That shows we have 28267 missing files. Something isn't right here. root@arcas [/x1/flex-jira-dance/jira/logs ] cat catalina.out| grep 'FLEX/FLEX-' | head -n3 2013-02-04 10:32:23,283 JiraTaskExectionThread-2 WARN pctony 632x30x1 pd7roa 128.232.134.143 /secure/admin/ProjectImportSelectProject.jspa [imports.project.handler.AttachmentFileValidatorHandler] The attachment 'rotateBug.zip' does not exist at '/x1/flex-jira-dance/data/import/attachments/FLEX/ FLEX-75 /10000'. It will not be imported. 2013-02-04 10:32:23,284 JiraTaskExectionThread-2 WARN pctony 632x30x1 pd7roa 128.232.134.143 /secure/admin/ProjectImportSelectProject.jspa [imports.project.handler.AttachmentFileValidatorHandler] The attachment 'workaround.mxml' does not exist at '/x1/flex-jira-dance/data/import/attachments/FLEX/ FLEX-305 /10001'. It will not be imported. 2013-02-04 10:32:23,284 JiraTaskExectionThread-2 WARN pctony 632x30x1 pd7roa 128.232.134.143 /secure/admin/ProjectImportSelectProject.jspa [imports.project.handler.AttachmentFileValidatorHandler] The attachment 'BusyCursorTest.zip' does not exist at '/x1/flex-jira-dance/data/import/attachments/FLEX/ FLEX-313 /10002'. It will not be imported. That is 3 missing files at the start of the import. root@arcas [/x1/flex-jira-dance/jira/logs ] ls -lah /x1/flex-jira-dance/data/import/attachments/FLEX/ FLEX-75 / total 628K drwxr-xr-x 2 root root 4.0K Feb 4 09:52 . drwxr-xr-x 12686 root root 356K Feb 4 09:56 .. -rwxr-xr-x 1 root root 261K Feb 4 09:52 10186 root@arcas [/x1/flex-jira-dance/jira/logs ] ls -lah /x1/flex-jira-dance/data/import/attachments/FLEX/ FLEX-305 total 368K drwxr-xr-x 2 root root 4.0K Feb 4 09:51 . drwxr-xr-x 12686 root root 356K Feb 4 09:56 .. -rwxr-xr-x 1 root root 1.8K Feb 4 09:51 10187 root@arcas [/x1/flex-jira-dance/jira/logs ] ls -lah /x1/flex-jira-dance/data/import/attachments/FLEX/ FLEX-313 total 380K drwxr-xr-x 2 root root 4.0K Feb 4 09:53 . drwxr-xr-x 12686 root root 356K Feb 4 09:56 .. -rwxr-xr-x 1 root root 14K Feb 4 09:53 10188 As you can see none of the blob objects in the zip file match. None. Did you do some kind of transforms on this data? The file that the import is looking for is taken from the xml import file (in the zip). Did you import the data into another version of jira to export only the FLEX project, i.e. filter out others. If so you have to import the attachments into that instance as they will get renumbered starting at 10000 - if not the attachments we have are from the original and mismatch your exported zip/xml file. As it stands I cannot use the attachments. The attachments on disk do not use files names as per the source file. They are stored as blobs on disk starting at 10000, and the filename is stored in SQL. Alex, as far as I can tell. You did the export from Adobe, into a staging instance and then exported again after some filtering I suspect. That will only work if you import the the files as part of your import/export process. So far I have spent over 75 hours on this since the start of the year. Is there anyway we can start the process again? You get a clean dump and attachments? If so please can we get it as a tar.gz ? I hope this all makes sense. If not just shout. {QUOTE}
        Hide
        Tony Stevenson added a comment -
        There has been significant offline activity in testing this. Communication with Alex has resulted in a new tarball for the attachments, and a new xml export file are being produced.
        Show
        Tony Stevenson added a comment - There has been significant offline activity in testing this. Communication with Alex has resulted in a new tarball for the attachments, and a new xml export file are being produced.
        Hide
        Tony Stevenson added a comment -
        Alex, once you have given us the nod I will need to move the current FLEX project to a new location (FLEX2 or something). So I can import old FLEX, and then move the new tickets from FLEX2 -> FLEX .

        This will mean preventing access to the project. Can you please confirm that this is ok too. I suspect the work will take hours, and will require a re-index at the end too.
        You may want to liaise with your community as to when this is best done. Please note that I am not likely available over this weekend or the next. Sorry.
        Show
        Tony Stevenson added a comment - Alex, once you have given us the nod I will need to move the current FLEX project to a new location (FLEX2 or something). So I can import old FLEX, and then move the new tickets from FLEX2 -> FLEX . This will mean preventing access to the project. Can you please confirm that this is ok too. I suspect the work will take hours, and will require a re-index at the end too. You may want to liaise with your community as to when this is best done. Please note that I am not likely available over this weekend or the next. Sorry.
        Hide
        Adobe JIRA added a comment -
        Hi Tony,

        Sure, I can look at the test instance. 20 missing attachments is not a
        problem, especially if you know which issues they are. We can manually
        re-attach those from the Adobe instance.

        Thanks,

        --
        Alex Harui
        Flex SDK Team
        Adobe Systems, Inc.
        http://blogs.adobe.com/aharui

        Show
        Adobe JIRA added a comment - Hi Tony, Sure, I can look at the test instance. 20 missing attachments is not a problem, especially if you know which issues they are. We can manually re-attach those from the Adobe instance. Thanks, -- Alex Harui Flex SDK Team Adobe Systems, Inc. http://blogs.adobe.com/aharui
        Hide
        Tony Stevenson added a comment -
        Alex,

        I have a test instance up and running which has claimed to have imported your data, with attachments. I am now upgrading this to be in line with production JIRA, and will then export the data to import into a project on production JIRA, which will mean moving the existing project out of the way, to do the import. Once imported we can move all *new* issues back into the FLEX project again.

        Can I ping you an email with the test instance URL so you can test some sample issues please? The import complained about 20 missing files from the attachments folder. I have exploded it three times now without error so I can only assume these are not present from source. If they are hyper critical we will need to start again.
        Show
        Tony Stevenson added a comment - Alex, I have a test instance up and running which has claimed to have imported your data, with attachments. I am now upgrading this to be in line with production JIRA, and will then export the data to import into a project on production JIRA, which will mean moving the existing project out of the way, to do the import. Once imported we can move all *new* issues back into the FLEX project again. Can I ping you an email with the test instance URL so you can test some sample issues please? The import complained about 20 missing files from the attachments folder. I have exploded it three times now without error so I can only assume these are not present from source. If they are hyper critical we will need to start again.
        Hide
        Tony Stevenson added a comment -
        Sorry,

        This slipped through my list, partly due to it seems infraabot not showing filters properly atm, and partly due to this being such a bloody long way down the list.
        Alex. I will re-investigate now.

        Apologies for the cock-up, entirely my fault.
        Show
        Tony Stevenson added a comment - Sorry, This slipped through my list, partly due to it seems infraabot not showing filters properly atm, and partly due to this being such a bloody long way down the list. Alex. I will re-investigate now. Apologies for the cock-up, entirely my fault.
        Hide
        Joe Schaefer added a comment -
        I'll bring it up with Tony unless
        there's some other action you want
        me to take.





        Show
        Joe Schaefer added a comment - I'll bring it up with Tony unless there's some other action you want me to take.
        Hide
        Sam Ruby added a comment -
        ping?
        Show
        Sam Ruby added a comment - ping?
        Hide
        Marcus Fritze added a comment -
        Any progress on this issue? The attachments are still missing in the Flex JIRA.
        Show
        Marcus Fritze added a comment - Any progress on this issue? The attachments are still missing in the Flex JIRA.
        Hide
        Gavin added a comment -
        ping pctony
        Show
        Gavin added a comment - ping pctony
        Hide
        Alex Harui added a comment -
        After JIRA gets upgraded, would that be a good time to fix this issue?
        Show
        Alex Harui added a comment - After JIRA gets upgraded, would that be a good time to fix this issue?
        Hide
        Alex Harui added a comment -
        Last Bug in the import was FLEX-33058.
        Show
        Alex Harui added a comment - Last Bug in the import was FLEX-33058 .
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Alex, it looks like we have an issue here. JIRA no longer allows you to associate attachments after import. They must be specified at import time. As per https://confluence.atlassian.com/display/JIRA/Restoring+a+Project+from+Backup#RestoringaProjectfromBackup-specifybackup -- To fix this, we will need to re-import all the jira issues again, and start over. Can you please confirm the last jira issue number that was
        Show
        #asfinfra IRC Bot added a comment - <pctony> Alex, it looks like we have an issue here. JIRA no longer allows you to associate attachments after import. They must be specified at import time. As per https://confluence.atlassian.com/display/JIRA/Restoring+a+Project+from+Backup#RestoringaProjectfromBackup-specifybackup -- To fix this, we will need to re-import all the jira issues again, and start over. Can you please confirm the last jira issue number that was
        Hide
        Alex Harui added a comment -
        Hi Tony, unfortunately, a sampling of issues shows that none of the attachments are linked from the issues. For example, I looked at FLEX-14619, FLEX-26916, FLEX-25951 and FLEX-28295. Their corresponding bugs in Adobe JIRA have attachments, and in the attachment folder I zipped up there are folders for these for issues with files in them.

        Can you verify the zip as you received it has folders for these issues? Maybe zip file got unzipped one folder deeper than it should have?
        Show
        Alex Harui added a comment - Hi Tony, unfortunately, a sampling of issues shows that none of the attachments are linked from the issues. For example, I looked at FLEX-14619 , FLEX-26916 , FLEX-25951 and FLEX-28295 . Their corresponding bugs in Adobe JIRA have attachments, and in the attachment folder I zipped up there are folders for these for issues with files in them. Can you verify the zip as you received it has folders for these issues? Maybe zip file got unzipped one folder deeper than it should have?
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Alex and co. Your attachments are in place, and I have re-indexed JIRA. So, have at it and enjoy!!
        Show
        #asfinfra IRC Bot added a comment - <pctony> Alex and co. Your attachments are in place, and I have re-indexed JIRA. So, have at it and enjoy!!
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Alex, I somehow lost your mail with the URL on it, I have just found it thanks to joe's ping reminder, I am now downloading the file from dropbox. In future, can we please not use dropbox? It is a bitch to use from the command line. Cheers. I'll update this once the attachments are there.
        Show
        #asfinfra IRC Bot added a comment - <pctony> Alex, I somehow lost your mail with the URL on it, I have just found it thanks to joe's ping reminder, I am now downloading the file from dropbox. In future, can we please not use dropbox? It is a bitch to use from the command line. Cheers. I'll update this once the attachments are there.
        Hide
        Joe Schaefer added a comment -
        Ping? Can this ticket be closed out or is there more work to do here?
        Show
        Joe Schaefer added a comment - Ping? Can this ticket be closed out or is there more work to do here?
        Hide
        Alex Harui added a comment -
        I posted a Mac Finder zip at a URL I sent privately. The MD5 is: 6dcbfb9c8185620eac479d2babe6426d
        Show
        Alex Harui added a comment - I posted a Mac Finder zip at a URL I sent privately. The MD5 is: 6dcbfb9c8185620eac479d2babe6426d
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Alex, as soon as you can get me a gzipped tarball (preference) or other linux compat zip file I will get the attachments in place. I will then need to kick off a re-index again too.
        Show
        #asfinfra IRC Bot added a comment - <pctony> Alex, as soon as you can get me a gzipped tarball (preference) or other linux compat zip file I will get the attachments in place. I will then need to kick off a re-index again too.
        Hide
        Tony Stevenson added a comment -
        The data has been imported, however as per email to Alex earlier today the attachments file is corrupted, and cannot be exploded. Yet the MD5sum matches too.
        Show
        Tony Stevenson added a comment - The data has been imported, however as per email to Alex earlier today the attachments file is corrupted, and cannot be exploded. Yet the MD5sum matches too.
        Hide
        Alex Harui added a comment -
        Hi,

        The projects statistics match what I had in the JIRA instance I used to create the import file, and a sampling of a dozen or so issues show they have the same content (minus attachments which I assume you will include in the real import). I did notice that some of the links in the description field of old bugs like FLEX-102 didn't come across cleanly. In my JIRA instance the URL is hidden and some nice text is shown as the link, in the test instance the URL is shown and the link is not active. Links in the comments field seem ok.

        If you have a few minutes to look into it, great, but if you don't, I think we can live with it since most folks won't need to access those links.

        I don't know if I get to speak for the entire PPMC, but I approve you to try importing with attachments into the real Apache JIRA instance.
        Show
        Alex Harui added a comment - Hi, The projects statistics match what I had in the JIRA instance I used to create the import file, and a sampling of a dozen or so issues show they have the same content (minus attachments which I assume you will include in the real import). I did notice that some of the links in the description field of old bugs like FLEX-102 didn't come across cleanly. In my JIRA instance the URL is hidden and some nice text is shown as the link, in the test instance the URL is shown and the link is not active. Links in the comments field seem ok. If you have a few minutes to look into it, great, but if you don't, I think we can live with it since most folks won't need to access those links. I don't know if I get to speak for the entire PPMC, but I approve you to try importing with attachments into the real Apache JIRA instance.
        Hide
        Tony Stevenson added a comment -
        Hi,

        Sorry for the delay, the underlying issue we were having has been relieved, or at least it looks like we have. I have managed to import the FLEX JIRA data into the test instance.
        Can you, the PPMC look at this "https://issues.apache.org/jira-test/browse/FLEX" and make sure it looks ok.

        If the PPMC gives the go ahead, I will need to rename the current FLEX project, and then import, and then move the current FLEX issues back into the FLEX project.
        Show
        Tony Stevenson added a comment - Hi, Sorry for the delay, the underlying issue we were having has been relieved, or at least it looks like we have. I have managed to import the FLEX JIRA data into the test instance. Can you, the PPMC look at this " https://issues.apache.org/jira-test/browse/FLEX " and make sure it looks ok. If the PPMC gives the go ahead, I will need to rename the current FLEX project, and then import, and then move the current FLEX issues back into the FLEX project.
        Hide
        Justin Mclean added a comment -
        Any further progress on this? Is there anything else we (Apache Flex) can do to help and/or speed up this process?
        Show
        Justin Mclean added a comment - Any further progress on this? Is there anything else we (Apache Flex) can do to help and/or speed up this process?
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> if we knew then what we knew now perhaps I would have aksed you. As it happens this upgrade was far from a planned one.
        Show
        #asfinfra IRC Bot added a comment - <pctony> if we knew then what we knew now perhaps I would have aksed you. As it happens this upgrade was far from a planned one.
        Hide
        #asfinfra IRC Bot added a comment -
        <gmcdonald> no worries Alex, our upgrade has literally just happened
        Show
        #asfinfra IRC Bot added a comment - <gmcdonald> no worries Alex, our upgrade has literally just happened
        Hide
        Alex Harui added a comment -
        I will start working on it. I asked on 5/1 if our 4.x import file would be compatible. I could have had it ready if I knew it was required.
        Show
        Alex Harui added a comment - I will start working on it. I asked on 5/1 if our 4.x import file would be compatible. I could have had it ready if I knew it was required.
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Hi, I bet you thought we forgot about you, right? Well we haven't. We recently completed a rather painful upgrade to Jira 5.0.4 - and we are ready to try the import again. Do you think you can provide us with data from 5.0.4 so i can try the import ASAP, before then next upgrade comes along...
        Show
        #asfinfra IRC Bot added a comment - <pctony> Hi, I bet you thought we forgot about you, right? Well we haven't. We recently completed a rather painful upgrade to Jira 5.0.4 - and we are ready to try the import again. Do you think you can provide us with data from 5.0.4 so i can try the import ASAP, before then next upgrade comes along...
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Sort of. Yes. All the JIRA pain we have been enduring the past 3 days is because of our need to upgrade, and our attempt to move back to a Linux VM, with an Oracle JVM. So we can try again, and then better shout at Atlassian once we are back on what they call a supported platform. News to follow in the next few days.
        Show
        #asfinfra IRC Bot added a comment - <pctony> Sort of. Yes. All the JIRA pain we have been enduring the past 3 days is because of our need to upgrade, and our attempt to move back to a Linux VM, with an Oracle JVM. So we can try again, and then better shout at Atlassian once we are back on what they call a supported platform. News to follow in the next few days.
        Hide
        Justin Mclean added a comment -
        Any further progress on this?
        Show
        Justin Mclean added a comment - Any further progress on this?
        Hide
        Alex Harui added a comment -
        Thanks Tony,

        Will the Flex import file be compatible with JIRA 5?




        --
        Alex Harui
        Flex SDK Team
        Adobe Systems, Inc.
        http://blogs.adobe.com/aharui

        Show
        Alex Harui added a comment - Thanks Tony, Will the Flex import file be compatible with JIRA 5? -- Alex Harui Flex SDK Team Adobe Systems, Inc. http://blogs.adobe.com/aharui
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> So in theory we are only days away from deciding on upgrading the main JIRA, or deploying a standalone instance of JIRA 5, that will be used for Flex, xwork etc
        Show
        #asfinfra IRC Bot added a comment - <pctony> So in theory we are only days away from deciding on upgrading the main JIRA, or deploying a standalone instance of JIRA 5, that will be used for Flex, xwork etc
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Alex, we just upgraded JIRA to the latest 4.x version. I am currently working on a JIRA instance, which my testing shows the import should work.
        Show
        #asfinfra IRC Bot added a comment - <pctony> Alex, we just upgraded JIRA to the latest 4.x version. I am currently working on a JIRA instance, which my testing shows the import should work.
        Hide
        Alex Harui added a comment -
        Any update? Is it time to reconsider using the SOAP interface to trickle in issues?
        Show
        Alex Harui added a comment - Any update? Is it time to reconsider using the SOAP interface to trickle in issues?
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Progress has been slow. The platform from which we deliver this service directly contradicts what Atlassian say we should run it with. We are looking at options to perhaps make that not be the case.
        Show
        #asfinfra IRC Bot added a comment - <pctony> Progress has been slow. The platform from which we deliver this service directly contradicts what Atlassian say we should run it with. We are looking at options to perhaps make that not be the case.
        Hide
        Justin Mclean added a comment -
        I'm unable to view Atlasssian issue. Has any progress been made? If not any suggestions on how to resolve this issue?
        Show
        Justin Mclean added a comment - I'm unable to view Atlasssian issue. Has any progress been made? If not any suggestions on how to resolve this issue?
        Hide
        Tony Stevenson added a comment -
        I have just opened a support issue with atlassian -- https://support.atlassian.com/browse/JSP-120420
        Show
        Tony Stevenson added a comment - I have just opened a support issue with atlassian -- https://support.atlassian.com/browse/JSP-120420
        Hide
        Bertrand Delacretaz added a comment -
        ok - I didn't try with a multi-GB file, just wanted to make sure people were aware of that "public link" option
        Show
        Bertrand Delacretaz added a comment - ok - I didn't try with a multi-GB file, just wanted to make sure people were aware of that "public link" option
        Hide
        #asfinfra IRC Bot added a comment -
        <pctony> Bertrand, you try that with a multi-GB that always closed the session with about ~200MB remaining. Dont forget our infra doesnt have a GUI in many places. Using simple ssh/scp to a trusted location is usually far simpler for us. It also means that we may need to move the data multiple times. Just making it a little bit easier for us is nice ;)
        Show
        #asfinfra IRC Bot added a comment - <pctony> Bertrand, you try that with a multi-GB that always closed the session with about ~200MB remaining. Dont forget our infra doesnt have a GUI in many places. Using simple ssh/scp to a trusted location is usually far simpler for us. It also means that we may need to move the data multiple times. Just making it a little bit easier for us is nice ;)
        Hide
        Bertrand Delacretaz added a comment -
        Regarding issues getting from dropbox without a browser, I tried wget on one of my binary files and it works fine:

        wget http://dl.dropbox.com/u/715349/tmp/clutch.jpg

        The URL is the one I get from the "copy public link" option on that file in dropbox (local menu at the right of the filename) - I assume the same should work for Alex's zip file if he can provide a similar URL.
        Show
        Bertrand Delacretaz added a comment - Regarding issues getting from dropbox without a browser, I tried wget on one of my binary files and it works fine: wget http://dl.dropbox.com/u/715349/tmp/clutch.jpg The URL is the one I get from the "copy public link" option on that file in dropbox (local menu at the right of the filename) - I assume the same should work for Alex's zip file if he can provide a similar URL.
        Hide
        Alex Harui added a comment -
        MD5 for FlexSDKENTDMV41.zip is 47454c96fd37f69233fac5d7216e80c6
        Show
        Alex Harui added a comment - MD5 for FlexSDKENTDMV41.zip is 47454c96fd37f69233fac5d7216e80c6
        Hide
        Alex Harui added a comment -
        I made a backup of the successful import into 4.1. It is on dropbox as FlexSDKENTDMV41.zip. Please give it a try. If it also fails, it might be time to give up and use SOAP to import the issues into JIRA.
        Show
        Alex Harui added a comment - I made a backup of the successful import into 4.1. It is on dropbox as FlexSDKENTDMV41.zip. Please give it a try. If it also fails, it might be time to give up and use SOAP to import the issues into JIRA.
        Hide
        Alex Harui added a comment -
        Tony, I looked in the file and there is no issue with that key. What os are you using? I used windows 7. Is this error early or late in the import? I had it get stuck the first several times as well. The log would have an out of heap. It worked with mx512 but ran out of heap reindexing so I forced a reindex. That is in vmware on my mac. I was hoping you would have fewer memory issues. If it appears to be almost done when it gets stuck try forcing reindex

        Sent from my Motorola ATRIX™ 4G on AT&T


        -----Original message-----
        From: "Tony Stevenson (Commented) (JIRA)" <jira@apache.org>
        To: Alex Harui <aharui@adobe.com>
        Sent: Fri, Feb 10, 2012 20:33:04 GMT+00:00
        Subject: [jira] [Commented] (INFRA-4380) Import Flex JIRA


            [ https://issues.apache.org/jira/browse/INFRA-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205725#comment-13205725 ]

        Tony Stevenson commented on INFRA-4380:
        ---------------------------------------

        Alex,

        I have now tried importing this 11 times. Each of them hangs at exactly the same part.

        2012-02-09 20:18:38,416 ProjectImport: CreateIssues-6 ERROR pctony 1218x174x1 186tytb 86.6.154.14 /secure/admin/ProjectImportSummary.jspa [atlassian.jira.project.DefaultProjectManager] Existing issue found for key FLEX-33059. Incrementing key.

        The progress then stops. I have let the job run for 27 hours now, and no progress. At the moment I am at a loss as too what else I can try. I have bumped the CATALINA_OPTS to settings that just mad, and it still doesnt complete. Xms=768M, Xmx=4096, Permgen=1024M




        --
        This message is automatically generated by JIRA.
        If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
        For more information on JIRA, see: http://www.atlassian.com/software/jira


        Show
        Alex Harui added a comment - Tony, I looked in the file and there is no issue with that key. What os are you using? I used windows 7. Is this error early or late in the import? I had it get stuck the first several times as well. The log would have an out of heap. It worked with mx512 but ran out of heap reindexing so I forced a reindex. That is in vmware on my mac. I was hoping you would have fewer memory issues. If it appears to be almost done when it gets stuck try forcing reindex Sent from my Motorola ATRIX™ 4G on AT&T -----Original message----- From: "Tony Stevenson (Commented) (JIRA)" < jira@apache.org > To: Alex Harui < aharui@adobe.com > Sent: Fri, Feb 10, 2012 20:33:04 GMT+00:00 Subject: [jira] [Commented] ( INFRA-4380 ) Import Flex JIRA     [ https://issues.apache.org/jira/browse/INFRA-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205725#comment-13205725 ] Tony Stevenson commented on INFRA-4380 : --------------------------------------- Alex, I have now tried importing this 11 times. Each of them hangs at exactly the same part. 2012-02-09 20:18:38,416 ProjectImport: CreateIssues-6 ERROR pctony 1218x174x1 186tytb 86.6.154.14 /secure/admin/ProjectImportSummary.jspa [atlassian.jira.project.DefaultProjectManager] Existing issue found for key FLEX-33059 . Incrementing key. The progress then stops. I have let the job run for 27 hours now, and no progress. At the moment I am at a loss as too what else I can try. I have bumped the CATALINA_OPTS to settings that just mad, and it still doesnt complete. Xms=768M, Xmx=4096, Permgen=1024M -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
        Hide
        Tony Stevenson added a comment -
        Alex,

        I have now tried importing this 11 times. Each of them hangs at exactly the same part.

        2012-02-09 20:18:38,416 ProjectImport: CreateIssues-6 ERROR pctony 1218x174x1 186tytb 86.6.154.14 /secure/admin/ProjectImportSummary.jspa [atlassian.jira.project.DefaultProjectManager] Existing issue found for key FLEX-33059. Incrementing key.

        The progress then stops. I have let the job run for 27 hours now, and no progress. At the moment I am at a loss as too what else I can try. I have bumped the CATALINA_OPTS to settings that just mad, and it still doesnt complete. Xms=768M, Xmx=4096, Permgen=1024M

        Show
        Tony Stevenson added a comment - Alex, I have now tried importing this 11 times. Each of them hangs at exactly the same part. 2012-02-09 20:18:38,416 ProjectImport: CreateIssues-6 ERROR pctony 1218x174x1 186tytb 86.6.154.14 /secure/admin/ProjectImportSummary.jspa [atlassian.jira.project.DefaultProjectManager] Existing issue found for key FLEX-33059 . Incrementing key. The progress then stops. I have let the job run for 27 hours now, and no progress. At the moment I am at a loss as too what else I can try. I have bumped the CATALINA_OPTS to settings that just mad, and it still doesnt complete. Xms=768M, Xmx=4096, Permgen=1024M
        Hide
        Tony Stevenson added a comment -
        I have the updated XML files, and I am now running a test import, on our test instance. I'll pass you the URL just as soon as it is ready.
        Show
        Tony Stevenson added a comment - I have the updated XML files, and I am now running a test import, on our test instance. I'll pass you the URL just as soon as it is ready.
        Hide
        Alex Harui added a comment -
        The FlexSDKENTDMV.zip was updated in dropbox. The new MD5 is: 4759cb2c4faf6e52f6cc87c1680adb12

        Doing a re-import from Adobe JIRA to a 4.4.1 instance was not practical given current logistics. Instead I manually modified the file to change the references to build 664 to build 660. I was able to import it with attachments into a test 4.4.1 instance on my local machine. Please attempt an import with this new file.
        Show
        Alex Harui added a comment - The FlexSDKENTDMV.zip was updated in dropbox. The new MD5 is: 4759cb2c4faf6e52f6cc87c1680adb12 Doing a re-import from Adobe JIRA to a 4.4.1 instance was not practical given current logistics. Instead I manually modified the file to change the references to build 664 to build 660. I was able to import it with attachments into a test 4.4.1 instance on my local machine. Please attempt an import with this new file.
        Hide
        Tony Stevenson added a comment -
        Import failed.

        "This data appears to be from an older version of JIRA. Please upgrade the data and try again. The current version of JIRA is at build number '660', but the supplied backup file was for build number '664'."
        Show
        Tony Stevenson added a comment - Import failed. "This data appears to be from an older version of JIRA. Please upgrade the data and try again. The current version of JIRA is at build number '660', but the supplied backup file was for build number '664'."
        Hide
        Alex Harui added a comment -
        I will try to get it to work with 4.4.1. What error did you get so I will know we are getting the same results?
        Show
        Alex Harui added a comment - I will try to get it to work with 4.4.1. What error did you get so I will know we are getting the same results?
        Hide
        Bertrand Delacretaz added a comment -
        I just chatted with Tony and it seems like he'll be able to keep the existing FLEX issues, and add the ones that will be imported into that same project.
        Show
        Bertrand Delacretaz added a comment - I just chatted with Tony and it seems like he'll be able to keep the existing FLEX issues, and add the ones that will be imported into that same project.
        Hide
        Tony Stevenson added a comment -
        As expected the editing the XML data and performing a manual search and replace on the data is not going to be a good idea, as there are hundreds of string matches, and we will inevitably replace other valid data.
        So it looks like we will need to get a 4.4.1 export from you, please.

        We do not need the attachments again, just the XML/ZIP data.
        Show
        Tony Stevenson added a comment - As expected the editing the XML data and performing a manual search and replace on the data is not going to be a good idea, as there are hundreds of string matches, and we will inevitably replace other valid data. So it looks like we will need to get a 4.4.1 export from you, please. We do not need the attachments again, just the XML/ZIP data.
        Hide
        Tony Stevenson added a comment - - edited
        and it failed.....

        The export was from a newer version of JIRA. The import process, as previously explained, is extremely version sensitive. The ASF uses v4.4.1 (build 660) -- You provided a backup from 4.4.4 (build 664) which will not import cleanly.
        I am trying to manually edit the XML, but this is far from ideal. if someone can prepare a 4.4.1 export in the mean time, just in case this method fails (as I expect it too).
        Show
        Tony Stevenson added a comment - - edited and it failed..... The export was from a newer version of JIRA. The import process, as previously explained, is extremely version sensitive. The ASF uses v4.4.1 (build 660) -- You provided a backup from 4.4.4 (build 664) which will not import cleanly. I am trying to manually edit the XML, but this is far from ideal. if someone can prepare a 4.4.1 export in the mean time, just in case this method fails (as I expect it too).
        Hide
        Tony Stevenson added a comment -
        Data currently importing into the test instance.... (without attachments for now)
        Show
        Tony Stevenson added a comment - Data currently importing into the test instance.... (without attachments for now)
        Hide
        Tony Stevenson added a comment -
        Ok,

        I am battling the stupidity of drop box right now. I am having to pull the files in using links on another machine then will copy them over to the jail that jira runs on.
        I'll get the test instance up an running asap
        Show
        Tony Stevenson added a comment - Ok, I am battling the stupidity of drop box right now. I am having to pull the files in using links on another machine then will copy them over to the jail that jira runs on. I'll get the test instance up an running asap
        Hide
        Bertrand Delacretaz added a comment -
        It's ok for the import to overwrite the current FLEX issues, there's just a few of them that we can recreate manually later if needed. I'm attaching the XML dump of those.
        Show
        Bertrand Delacretaz added a comment - It's ok for the import to overwrite the current FLEX issues, there's just a few of them that we can recreate manually later if needed. I'm attaching the XML dump of those.
        Hide
        Alex Harui added a comment -
        Added MD5 checksums
        Show
        Alex Harui added a comment - Added MD5 checksums

          People

          • Assignee:
            Tony Stevenson
            Reporter:
            Alex Harui
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:

              Development