James Rowe
2016-01-28 22:53:59 UTC
Greetings everyone! I've been working with an MFI for sometime now, trying
to migrate their data from their old system on to the mifosx platform. I
have tried to migrate using the DataImportTool, but it doesn't support
everything we need to do when migrating loan transactions. So instead, I
wrote a simple python script to migrate the transaction history. It works
in small numbers, but the API seems to be extremely CPU intensive and for
larger loads we end up with max cpu usage on all of the cores. We decided
to run the migration on the largest AWS instance we could but its still
taking a long time on many CPUs Loading Image...
v
I'm looking into alternative ways to import the loan transaction history
into Mifosx, but I'm struggling to figure everything out. What exactly does
mifos do on loan transactions that makes it consume so much CPU? Is there a
way to directly dump transaction history into the database and still have
everything that mifos needs to run? I currently only use the API for
transaction history because I don't know how mifos stores transactions in
the database. I'm not sure what the m_loan_arrears_aging table does, and
what I should put in each of the _derived columns in the
m_loan_arrears_aging table and the m_loan_transaction tables. I'm guessing
those are all fields that Mifosx is calculating when a transaction is made
through the API, but for a migration they don't seem necessary. Are these
fields needed? Is it safe to leave them blank during migrations or fill
them in with dummy data? I feel that even if they are required, once I can
figure out what these fields stand for, I can figure out what is done on a
per transaction basis, then I can roll up transactions in script, and write
out the transactions to the database. Since its a migration, I know the
final state of each loan that will be migrated, so I imagine that if I skip
the recalculation every time, I can finish the migration much much faster.
Any advice would be greatly appreciated.
Thanks,
James
to migrate their data from their old system on to the mifosx platform. I
have tried to migrate using the DataImportTool, but it doesn't support
everything we need to do when migrating loan transactions. So instead, I
wrote a simple python script to migrate the transaction history. It works
in small numbers, but the API seems to be extremely CPU intensive and for
larger loads we end up with max cpu usage on all of the cores. We decided
to run the migration on the largest AWS instance we could but its still
taking a long time on many CPUs Loading Image...
vI'm looking into alternative ways to import the loan transaction history
into Mifosx, but I'm struggling to figure everything out. What exactly does
mifos do on loan transactions that makes it consume so much CPU? Is there a
way to directly dump transaction history into the database and still have
everything that mifos needs to run? I currently only use the API for
transaction history because I don't know how mifos stores transactions in
the database. I'm not sure what the m_loan_arrears_aging table does, and
what I should put in each of the _derived columns in the
m_loan_arrears_aging table and the m_loan_transaction tables. I'm guessing
those are all fields that Mifosx is calculating when a transaction is made
through the API, but for a migration they don't seem necessary. Are these
fields needed? Is it safe to leave them blank during migrations or fill
them in with dummy data? I feel that even if they are required, once I can
figure out what these fields stand for, I can figure out what is done on a
per transaction basis, then I can roll up transactions in script, and write
out the transactions to the database. Since its a migration, I know the
final state of each loan that will be migrated, so I imagine that if I skip
the recalculation every time, I can finish the migration much much faster.
Any advice would be greatly appreciated.
Thanks,
James