Monday, April 11, 2011

New Version of Money Scripts Posted

The production version of the scripts has been updated to the latest snapshot.  Thank you to everyone for testing!  There are still "opportunities for improvement", both for the scripts and for the installation/configuration help files.  I need to summarize all of the optional configuration parameters now available in sites.dat, and add to the configuration page.

You can probably guess that my time has been limited for this project, since I haven't posted much lately.  I don't see that changing through the summer months, but I'm here and will fix/add things as I can.  I expect to be mostly unavailable from around mid-May through the month of June, but will check in to see if any bad things have been reported.

225 comments:

  1. Hi
    I've followed your site over the last few days and tried to adopt my money 2004 program to work accepting stock and OEIC fund data. Most of my stocks are on the yahoo site so it looks promising,I'm UK based, however when it imports into money I struggle to get the prices to update. The program wants to wipe out all my stocks and therefore portfolio and that seems the only option to me. Where could i be going wrong?

    ReplyDelete
  2. Thanks Robert (and Cal) for all the work you've been doing. I just downloaded this latest production version and note one glitch and some differences.

    The new glitch (this did NOT occur with the previous version): The "sending results to Money" section seems to choke if there is a "(" in the file name. I have messages:

    Sending results to Money...
    '.\xfr\CHASE_' is not recognized ...

    In each case, the underscore is followed by a "(". I'd sure appreciate it if you could fix this.

    I also see messages for downloads from Chase, USAA and Fidelity stating that:

    +Scrubber: Updating null time values.

    What does this mean?

    Finally, there is a message following some Fidelity account downloads stating:

    OFX statement is missing required section.
    ** Review .\xfr\FIDELITY20110414091004.ofx for possible clues...

    These are all investment accounts with no securities in the account -- only cash. And, unlike the Vanguard problem, Fidelity downloads have never bombed Money. Are these accounts still getting updated?

    Thanks.

    Thanks.

    ReplyDelete
  3. Maurice:

    I'm not familiar with the Money 2004 software, and don't know much about UK configurations. Maybe a UK user can help shed some light...?

    ReplyDelete
  4. Ron:

    1. To verify, you have a "(" character in your sitename? The latest version uses os.system() to send files, but this function doesn't support all Windows filename characters. I didn't consider anyone using parentheses in their sitename, but will add the "()" chars to the filter.

    2. The scrubber routine is now "verbose", telling you things it didn't previously. I can add a non-verbose user flag, but I feel it was better to show what the script is doing.

    3. The Fidelity error is probably related to the script finding an invposlist section, but not the related seclist. As there are no investments, that makes sense. I can see if I can find a logical method to pass cash-only accounts that fail this test. In the meantime, I would suspect that it will work correctly when there is an actual transaction (i.e., posting interest to the account), since that should contain a seclist entry for the money market fund, or whatever it's stored in. So... it may not matter as-is?

    ReplyDelete
  5. Robert,

    I do indeed have "(" in the sitename (and, of course a ")"). It was a helpful (to me) naming method where I have multiple accounts at the same institution.

    I prefer the verbose scrubber; just didn't know what it was doing.

    I'll have to wait until the end of the month to see if the interest gets posted. But I do have one Fidelity account that is "just cash", but the cash is in the Fidelity Municipal Money Market fund -- that one does not show an error even without interest; and there is a position in FTEXX. It's some others that show the error.

    If I double click on one of the OFX file that Pocket Sense shows this error, it will import into Money with no issues; so it's probably not just the absence of in the Vanguard accounts that is causing that problem.

    Thanks again for all your work on this.

    -- Ron

    ReplyDelete
  6. Still using the October versions of the scripts (haven't had time to update myself). Using Money Sunset

    Something strange started happening about 3 weeks ago. All three of my 401(k)'s (2 with Fidelity, 1 with TRowe Price) are having an issue. As far as I can tell the OXF files have all the buy transactions in them. However, when they get imported into Money, the transactions are ignored, and Money just tells me the positions don't match and asks to add the new units.

    All three were working fine with the scripts previously. Any one experiencing the same thing, or have any suggestions?

    ReplyDelete
  7. Carlos: Not that it will help, but I highly recommend installing the latest scripts. Some of the changes were new additions, but there were also a number of bug patches.

    Are you certain the transactions are missing in Money? I've seen the same with Vanguard before. Transactions import incorrectly, then get ignored. By default, Money ignores duplicate transactions, or those that it "thinks" are duplicates. If I delete transactions in Money and then repeat, it accepts them again.

    ReplyDelete
  8. Robert - I installed the new scripts and I still have the problem (worked great in all other aspects). Bizarre that it is only happening on the 3 401(k)s (and in all 3) Transactions are definitely not in Money. And Money recognizes the positions don't match (no doh! it is missing the buy transactions) and helpfully volunteers to add shares.

    ReplyDelete
  9. I tried some additional debugging, to no avail. I changed the account number in the OXF to force a "remarrying" of the file to the account. Money again told me that the positions didn't match my account and offer to update them, but ignored all of the transactions in the file.

    I then changed the account number again, and this time I married it to a brand new virgin account. Again, none of the transactions were imported, just the total positions.

    I'm looking at the OXF and I can definetly see the transactions:
    BUY


    My regular investment accounts seem to be updating fine (at least they did 3/31 when dividends were paid out). I just did a purchase in one of them, so I should know tomorrow if it downloads OK. But for now, it appears the problem is just with the 401(k) accounts.

    ReplyDelete
  10. I've also tried deleting the statement from Money. Deleting ALL transactions that are in that statement, and reimporting the statement. No love. All I get from Money is the notification that my positions don't match the statement that was downloaded. I'm running out of ideas.

    ReplyDelete
  11. Carlos: I don't download any 401K accounts myself, so I'm not sure what is different. Is it defined as a 401K in Money? Have you tried creating a non-401K investment account, and connecting the ofx download to it? Does it still skip transactions?

    ReplyDelete
  12. Latest (unsuccessful!) debugging:
    1) Edit the OXF file to change FITIDs, since clients are supposed to use those to identify transactions. No love.
    2) Created new investment account (not marked as retirement or 401k). Imported the statement. No luck. Only positions show up.
    3) To see if I was crazy - changed account number on one of my regular investment OXF files and matched it with the new investment account. Transactions imported!

    So - it appears to be something unique about the transactions in the bad OXF files. No clue what, though.

    ReplyDelete
  13. Carlos: Do you have any special settings in the sites.dat file for these accounts (appid, appver ... etc.?). Are there any scrubber messages for these files?

    ReplyDelete
  14. 401K is a later addition to the OFX specs, and I think it just provides value data, at least in some cases. I don't know why a 401K account would be this way, but it is common for 401K holders to get less detail than other types of holders. :-(

    (I had missed the roll-over to the new thread. I had thought things just got quiet).

    ReplyDelete
  15. Robert and Carlos....

    I also have a Fidelity account and while I was doing extensive tests using pocketsense and money in parallel, I came across the same problem.

    If I used money to update the account, the cash type transactions usually got marked as :Reinvest Interest or Dividend". When using pocketsense, It got flagged as a buy if I let money adjust positions. I would then manually change the buy to reinvest interest.

    I will try to go back and see if I can find the ofx input files I saved from each method. I am pretty sure that they were much the same or I would have alerted Robert to the difference. At the time, I attributed the difference to Money 2007.

    Ray

    ReplyDelete
  16. Robert,

    I don't know much about Python scripts, but is the process to "add the "()" chars to the filter" merely a matter of changing line 255 in ofx.py? That's the line that now reads:

    sitename = ''.join(a for a in sitename if a not in '&\/:*?"<>|')

    ReplyDelete
  17. Hmm. That first character in that line should be two single quotes separated by a space. When it got posted, the space got removed.

    ReplyDelete
  18. Ron: I made a couple of minor changes to the code and updated the "production" snapshot. Sitename entries with () chars should work.

    ReplyDelete
  19. Robert: Thanks. It works fine with the "()" chars now. After the end of the month, I'll let you know about the interest in the Fido accounts that are showing the no SECLIST error message.

    ReplyDelete
  20. any idea what's going on with AMEX accounts? They were working fine until yesterday. Now I get "An ERROR occured retrieving POST response from online.americanexpress.com
    Exception type:
    Exception Val: The read operation timed out

    ReplyDelete
  21. it didn't let me post the exception type.
    it is < class 'ssl.SSLError' >

    ReplyDelete
  22. Nevermind the previous 2 comments. Realized that there was a problem with the AMEX site itself and had nothing to do with the scripts.

    ReplyDelete
  23. I haven't had time to do additional debugging of the 401(k) issue; I'll post as soon as I have more information. I do recall that in the past, when Money updates were still alive, Money would occasionally miss transactions in the Fidelity 401(k) -- but NEVER in the T Rowe Price. The strange things is the PocketSense Scripts were working, and then stopped working.

    Cal - As far as I can tell the transaction history is in the OXF file. I will make a non-retirement investment purchase to see if I notice a difference.

    The good news, is since the data is there we may be able to scrub it!

    ReplyDelete
  24. Used Money for about 15 years. Tried switching to Quickbooks - horrible, horrible, horrible, especially the customer service. Trying to use these scripts to get Money usable again.
    I have several investment accounts. I thought that if I used the scripts to update stock prices in a 'watch' account, that Money would automatically updates the values in the other accounts, but that does not appear to be happening. Is this the expected behavior?

    ReplyDelete
  25. Oops - that should say Quicken, not Quickbooks.

    ReplyDelete
  26. Keith: If the symbols match between the two, then the prices will update globally. Keep in mind, however, that the scripts can only provide one quote update per symbol per day. Once Money has received one for a given date, no more will be accepted for that date.

    A few issues that we've seen:

    1. Yahoo symbol different than Money symbol.
    2. Money configured for one currency, but Yahoo quotes being collected in another. Money prefixes all quotes with a currency tag, but it's hidden when "the same as the default setting in Money". If you only work in US Dollars, this is not an issue.

    ReplyDelete
  27. I think I just stumbled onto something. You mention one quote update per symbol per day. I think this is what's causing the problem, even though I'm *running* the update on a different day. Because the markets were closed Friday, and today is a Saturday, the download script is still reporting the quote date as 4/21. My money file already has entries from 4/21 where I was testing out the script in the middle of the day.

    Yep, that's what is happening. I just manually deleted the 4/21 price for one of the stocks and re-ran the test quote download. It updated my Money file to include the 4/21 closing price for this stock.

    Thanks
    Keith

    ReplyDelete
  28. Robert and Carlos....

    I too have the same problem with Fidelity 401k. A friend of mine has both 401K and non 401K accoutns with Fidelity. The problem seems to be with Fidelity 401K accounts. As someone reported, all the data is there in the OFX file. All the same THANKS very much for putting this all together.

    ReplyDelete
  29. Robert,

    Today I had the same problem as I reported before with some Fidelity downloads that, this time, DID have transactions.

    However, in the one case, a SECLIST section was created and PS allowed Money to import the OFX file. In the other, PS did NOT allow Money to import the OFX file, but clicking on the OFX file directly resulted in a proper import into Money.

    Looking at the OFX file, I see that in the one that PS accepted, there is a SECLIST section and the transaction appears to be a dividend. In the other, the transaction appears to be interest. In neither file are there any positions in the INVPOSLIST section.

    ReplyDelete
  30. If I had a sample 401K OFX file, I could probably write a scrubber to make Money accept it. The sample should have at least one transaction, and the xxxxxxxxx (where xxxxxxxxx represents the "account ID") should be replaced in all places with 123456789. I note that my Fidelity non-401k transactions now have the account ID also starting each string, so that those same 9 characters should be replaced with 123456789.

    I could write a little account number purger script in Python, but it should be easy enough if you have a simple text editor.

    Even with the account number gone, the file would identify the value of your 401K if connected to you. Changing quantity and value numbers without hosing up the file is tricky.

    ReplyDelete
  31. Ron: If you look at the two files, would the following logic have allowed both to pass to Money correctly?

    Block if a INVPOS tag exists, but SECLIST does not.

    ReplyDelete
  32. If anyone wants to pass a 401k OFX file to Cal for testing, feel free to send to me and I'll pass it along. pocketsense1 /at\ gmail /.\ com. As Cal mentioned, clean it of personally identifiable data (account numbers, etc.).

    ReplyDelete
  33. Robert: I think that logic is what PS is doing now. The file with the INVPOS section and a SECLIST section passed to Money.

    The file with the INVPOS section and no SECLIST section did not, even though there was a valid transaction.

    ReplyDelete
  34. Ron: The current version looks for INVPOSLIST & SECLIST. You mentioned that there were no positions listed in INVPOSLIST, so that's where I was headed.

    ReplyDelete
  35. Robert:

    AHA! I was confusing INVPOS tag with INVPOSLIST tag.

    Yes, I think the logic you propose would work because, it is the case in both files that there is NO INVPOS tag; so if the logic checked that, rather than checking for the INVPOSLIST tag, PS should function OK on my system.

    ReplyDelete
  36. Beta posted with updated INVPOS / SECLIST filter. It would be nice if others can try it too, since it's a slightly different logic. I never felt great about the original. It was a guess / makeshift fix to a really bad problem... i.e., Money bombing. The INVPOS logic makes more sense.

    sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip

    ReplyDelete
  37. Robert: It seemed to work properly and did not block any uploads to Money. Of course, since I recently downloaded, there were no transactions in those accounts, so I couldn't do a "full" test. (There won't be any new transactions in those accounts until the end of the month).

    Thank you very much.

    -- Ron

    ReplyDelete
  38. Thanks Ron. I looked through some test files I have to verify the INVPOS/SECLIST logic for ofx messages I *know* bombed Money, and they would all be blocked correctly. I feel better...

    ReplyDelete
  39. Live production snapshot updated: 03-May-2011.

    ReplyDelete
  40. Thank you very much, Robert.

    ReplyDelete
  41. No doubt I am missing something. Assume I have no functional bank or investment accounts directly downloading OFX, because I live in the hinterland of Canada. All that I want to do is download stock quotes using yahoo.ca. So far I have everything set up - I can set up the stocks and mutual funds, and test works great. I can even send the data to Money after testing it, that works too. But there is something I'm missing on getting getdata.py going. What is the URL that you would enter for your dummy "Tickers" account (the URL is apparently mandatory), such that getdata.py actually works? Sorry if this is a dumb question; I do not claim to be adept at these things.

    ReplyDelete
  42. Hazelbrae, in Money create a new investment account. Name it Quotes, for example. I don't know what currency your quotes come in as, but the currency of that account should match that of the quotes. Specify that account as the place that the Quotes OFX file downloads to.

    ReplyDelete
  43. Hazelbrae: If I understand correctly, you have tested the quote download using the manual "test" feature in Setup.py? Did you "Enable Stock/Fund Quotes" in Setup? If they are enabled, then you should see "6. Disable Stock/Fund Quotes" on the menu. Getdata will only grab stock/fund quotes if the option is enabled.

    ReplyDelete
  44. Carlos and Robert, I just started using these scripts the evening of May 3, 2011, so got the latest release. Today I got around to adding a 2 month old Fidelity taxable account. There are a dozen bonds bought in this account a month ago. So for my first download, I expected to see a dozen purchases and a few interest payments. What appeared were the few interest payments, but all the bonds were added vs bought. Fixed my entries manually and went on. But hope to see a fix one day. I'm afraid I do not have the OFX file or would send to you. Even with this defect I'm loving these scripts. I had so many issues manually tweaking Quicken each week I'll live with this and hope a fix becomes obvious to you one day. Thanks!

    ReplyDelete
  45. The download interval defaults to 7 days with a new install, meaning the script requests transactions from the bank that posted within the past 7 days. Since it also grabs the statement summary, it saw you had positions (but no buy transactions)... so it added them automatically. To grab transactions from the past couple of months, set this parameter in sites.dat:

    defaultInterval: 60

    ReplyDelete
  46. I am not the previous poster, but

    My default interval IS 7 days, but my Fidelity interval is 30. It does not make any difference on my system either. The statement says I have two transactions, one a dividend, and another an intrest transaction. When they import into Money, they both showed up as Add Shares. I have been using these scripts for more than a year, and it has ALWAYS acted this way. I long ago surmised that it was a Money problem since the OFX file looks proper.

    Ray

    ReplyDelete
  47. Thanks Ray. I'd say you are correct. The bank sends the OFX file, and we send it to Money after scrubbing for known issues. We really have no direct control over how a specific provider packages their OFX statement, or how Money decodes it's meaning.

    ReplyDelete
  48. "I am not the previous poster, but" person,
    1. Is your Fidelity account a 401K? If so, there are known problems with that. See post by "I too have the same problem with Fidelity 401k" person.

    2.Are you using Money 2005 or a newer version such as Money Plus Deluxe Sunset?

    ReplyDelete
  49. Quote Updates: I installed the new version of the PocketSense scripts a couple of days ago and am trying to deal with some missing quotes. This is not a new issue but something I asked about in February 27 on this blog and got some advice on what to do that I am now trying to follow for the first time. My situation is that I have some Ameritrade accounts where some of the investments do not routinely get updated quotes when I download transactions. I do not have a separate Microsoft Money quote account but have the following parameters set in Sites.dat:
    SaveTickersFirst: Yes
    SaveQuoteHistory: No
    QuoteCurrency: USD
    EnableYahooScrape: Yes
    YahooTimeZone: -5:EST
    ShowQuoteHTM: YES
    AskQuoteHTM: NO
    This is for funds rather than bonds, and Yahoo seems to recognize all of the symbols when I look them up on finance.yahoo.com. I am not getting any error messages, but I also am not getting any updated quotes. Any advice?

    ReplyDelete
  50. I meant to say this is for funds rather than stocks. Is there a way to update my post?

    ReplyDelete
  51. Regarding the dialog box that money puts up after each OFX import, I finally got around to using Spy++ on it and determined what it needed to clear.
    Here's sources for a small app I wrote to watch for the dialog box and "push the button" over and over again until it can't find one for 90 seconds.
    It needs work to make it more configurable, but the core part is the FindWindow and PostMessage logic; if there is some python extensions to access these Windows API, then the script could manage it directly, otherwise, running an .EXE from this code before starting the Python script will do the trick.


    // MoneyImportResponder.cpp : Defines the entry point for the console application.
    //

    #include "stdafx.h"

    #define STR_MONEY_IMPORT TEXT("Import a file")
    #define INITIAL_WAIT 30
    #define INTERVAL_DELAY 2
    #define IDLE_TIMEOUT 90

    #define SEC_TO_MSEC(x) ((x) * 1000)
    #define MSEC_TO_SEC(x) ((x) / 1000)

    int _tmain(int argc, _TCHAR* argv[])
    {
    HWND hwndMoneyDlg;
    DWORD dwIdleStart;

    printf("Waiting %d seconds to start", INITIAL_WAIT);
    Sleep(SEC_TO_MSEC(INITIAL_WAIT));

    dwIdleStart = GetTickCount();
    while (GetTickCount() - dwIdleStart < SEC_TO_MSEC(IDLE_TIMEOUT))
    {
    printf ("Looking for Dialog: '%ls'...\r\n", STR_MONEY_IMPORT);
    hwndMoneyDlg = FindWindow(NULL, STR_MONEY_IMPORT);
    if (hwndMoneyDlg != NULL)
    {
    printf("Dialog found!\r\n");
    PostMessage(hwndMoneyDlg, WM_COMMAND, 2, 0); // Simulate pressing OK button
    dwIdleStart = GetTickCount();
    }
    else
    {
    printf("No dialog found. Idle %d\r\n", (DWORD)(MSEC_TO_SEC(GetTickCount() - dwIdleStart)));
    }
    printf ("Waiting %d seconds\r\n", INTERVAL_DELAY);
    Sleep(SEC_TO_MSEC(INTERVAL_DELAY));
    }

    return 0;
    }

    ReplyDelete
  52. PHWhite:

    1. Enable stock/fund quotes in Setup.
    2. Create a quote account, and associate the stock/fund quotes with it on first import.
    3. Make sure the Yahoo symbols and currency match the Money symbols and currency. If not, you can define an alternate symbol when passing the results to Money. See the script revision history (http://sites.google.com/site/pocketsense/home/msmoneyfixp1/revisionhistory) for a description of how this works.

    ReplyDelete
  53. Thanks for the C code George. Something to play with, and possibly integrate into the scripts :)

    ReplyDelete
  54. Until Robert folds the code into PocketSense, Hung Le has incorporated the Java equivalent of George's code into his Java app that will auto-click the Import dialogs from PocketSense into Money as long as the app is open. It works great and I encourage Robert to incorporate George's code. See http://microsoftmoneyoffline.wordpress.com/2010/02/12/java-app-to-update-quotes/#comment-842 for more.

    ReplyDelete
  55. Quote Updates: I seem to have gotten this working but I live in the Pacific timezone so am wondering whether I should use the default YahooTimeZone: -5:EST. Other than this question everything was straight forward. Thanks for the advice.

    ReplyDelete
  56. Credit Card Transaction Download Options: I have a Capital One Mastercard and a Visa credit card issued by US Bank. Capital One offers OFX file downloads if you log in to the Capital One web site but does not support OFX file downloads for financial management software. US Bank also offers OFX file downloads by logging in but has a monthly charge for OFX file downloads via financial management software. Are there any reasonable free options for transaction downloads for these two credit cards?

    ReplyDelete
  57. PHWhite: Try it with the -5:EST setting and see if your quotes post with the right dates (i.e., the same dates shown on the Yahoo Finance site itself). Can't help on the credit card issue.

    ReplyDelete
  58. I'm the May 11, 2011 6:11pm poster. FYI, I had the interval set for 150 days. Noticed the later posts that day indicating this may be a Money issue. If/when I see this again I'll take a closer look at the OFX file.

    On another note I've had a couple of interesting experiences with quotes/updates that someone else may appreciate. I had one "stock" symbol classified as a "mutual fund" in the MS-Money Details section for that symbol. Additionally, had one "mutual fund" symbol classified as a "stock" too. You have to get that straight in order to receieve quotes/updates properly. One of these instances caused a "no define transaction" to be added to the account as the security name, but without a symbol, and the quoted price was for the poor stock that was classified as a mutual fund. Fixed by simply changing security type in the details section. It's embarrassing but easily remedied.

    ReplyDelete
  59. I've been absent, but now that I'm checking in I see that there haven't been any comments in a long while. I take that to be a good thing.

    ReplyDelete
  60. I have a Capital One direct banking account. It was working fine till the last few days. Now everytime I download the OFX, I get a general error message

    ReplyDelete
  61. Robert,
    Thank you so very much for this great product. I have been a MS Money user forever, with account transactions dating back to 2001. This download process works perfectly and I could not be happier!!!

    I do have one question though regarding the "updated" date that shows in the "Favorite Accounts" list. This date was updated by MS Money whenever it did the automated update. Is there anyway that the data upload can update that date to show when the last update was done?

    Again, great work - very impressive indeed. I also saw in your base profile that you are from TN - What part? I was raised just north of Nashville, but live in California now.

    Regards,

    Bob H.

    ReplyDelete
  62. Bob: We don't have access to specific parameters/dates/lists inside Money (e.g., "favorite accounts"). The scripts use the Money OFX import routines, but we have no way to directly control how the "importer" handles the data internally.

    I'm located in East TN... near Knoxville. I appreciate the positive feedback!

    ReplyDelete
  63. Has anyone else been getting garbled quotes? I believe it started last Friday for me and now I get random trashed quotes dated 1/1/1970 on two or three of my downloded stocks.

    ReplyDelete
  64. Thanks for the catch... yes, some of my quotes are garbled too. I'll check into it.

    ReplyDelete
  65. Just ran a test with inconsistent results. 2 out of 3 identical requests to the Yahoo server for their own symbol (YHOO) returned "correct" data, and one returned "junk". Definitely on the Yahoo end... so we'll wait to see if they "fix" it.

    ReplyDelete
  66. I haven't seen the quote errors in a while. Assuming it was a temporary glitch with Yahoo (?).

    ReplyDelete
  67. Bob,
    Hope you're havin' a good summer. Yes, Yahoo has been fixed. Here's a summary of that and other recent developments this month...

    http://microsoftmoneyoffline.wordpress.com/2011/07/28/erroneous-yahoo-quotes-making-qfx-files-work-with-money/

    -Dan (ameridan)

    ReplyDelete
  68. Well... I finally got around to replacing the old XP home computer. The new one is running Windows 7 64 bit, so I was curious as to how Money would run. I downloaded the Money Sunset edition and the install went fine. When first starting, it threw up a dialog in a foreign language (or something that appeared Arabic), which I couldn't read. I assume that this was some type of "agreement" to use the software without warranty. I clicked on the left button (again, I couldn't read the two... but figured it was a 50/50 shot at being "agree"). It was the correct pick. The Money screen appeared in fine English.

    I installed the 64 bit version of ActivePython 2.7, and tested the Pocketsense Setup routine. It worked. Lastly, I tested Getdata on the new system with my old files. It seemed to work as expected, so all looks well (so far). Just thought I'd post a status report for my initial venture into using the scripts with Money Sunset on a 64 bit Windows 7 machine.

    ReplyDelete
  69. Good to hear! Seems like I recall 64-bit issues for many Money users, so perhaps it has more to do with administrative permissions.

    ReplyDelete
  70. Re: admin permissions. That's possible. I have configured this machine to operate with "least user privs", even for the account that I personally use for routine work. The install was performed with admin priv (of course), but the actual "use of Money" is being performed with minimal privs.

    I suspect that many folks don't have a firm grasp of user privs and how the OS handles those (?), so that could lead to issues. So long as they put the scripts and their Money data file into their own "my documents" tree, they should be fine. Otherwise, they would need to assign permissions to wherever else they chose to put it. My opinion is that it is best to keep with the Windows default user tree structure whenever possible, rather than trying to add unique root trees (folders). For Vista and Win7, this means placing personal files (including the Money data file and PocketSense scripts) somewhere into their \Users\UserName\... tree.

    ReplyDelete
  71. I should have also posted: I've been using the system for a few more days now, and have used most features of Money, just to play around. Everything has worked well so far. The scripts are working exactly as they should, so it's looking good.

    ReplyDelete
  72. Hi, I'm from Sri Lanka & does not have much knowledge on programming. I use Microsoft money & used to update my stock prices manually as CSE (Colombo Stock Exchange) is not available with MSN finance.

    Since ft.com has CSE listed (ex - NTB.N0000:CSE) I would like to update stock prices in Money automatically via ft.com site.

    Can someone please help me with the coding?

    ReplyDelete
  73. Watch your data for Dow Jones Industrials from YaHoo. Looks like it is messed up/missing again!

    Ray

    ReplyDelete
  74. ^DJI2MN seems to be working for the Dow Jones. Who knows how long it will work for....

    ReplyDelete
  75. Just how messed up is YaHoo!

    Yes, ^DJI2MN returns the quote with the ticker symbol of ^DJI2MN. Thanks for that Carlos.

    ^DJI or should I say INDU is also working correctly???? at this time. If you request ticker symbol ^DJI, YaHoo returns nothing, BUT if you request INDU, Yahoo returns the quote with the ticker symbol of ^DJI. Seems a little confusing.

    Makes one wonder what ticker symbol ^DJI2MN really means.

    Either way, I am glad I do not have to rely on YaHoo at this time. I just keep testing this software for the day I may really need it. Keep up the good work Robert.

    Ray

    ReplyDelete
  76. That is DJI 2-minute update. I think that means that it is not realtime, and it may or may not be further delayed by Yahoo. I don't know. If it mattered, I would try it out during market hours.

    Cal Learner

    ReplyDelete
  77. Trying to set up accounts for Boeing Employees Credit Union, I get "BECU does not have a BANKID value defined." error. But no bankid is given on the OFX site. How do I find it?

    ReplyDelete
  78. This works for me...

    SiteName : BECU
    AcctType : BASTMT
    fiorg : becu
    fid : 325081403
    url : https://www.becuonlinebanking.org/scripts/serverext.dll
    bankid : 325081403
    brokerid :
    appid :
    appver :
    mininterval:

    ReplyDelete
  79. When I try it with the fid you give, I get:

    "local variable 'query' referenced before assignment
    An online error occurred when testing the new account."

    When I try it with the fid I find in a downloaded .ofx from my bank, I get an error, and looking in the xfr folder and examining the .ofx there, I find, "The user cannot signon because of invalid user ID or password."

    In both cases, I am using the identical user ID and password I use to sign on and download manually. (I triple-checked this.)

    Any ideas what I should try next?

    ReplyDelete
  80. IIRC, there was a detail about reformatting your password entry format: if you use "PassWord" try "password" ;)

    ReplyDelete
  81. I converted George's code to Python as he suggested. It adds logging to a file named PopDown1.log to the current directory. I had tried do also handle the error pop-up, but that is a problem; it's title bar says "Microsoft Money" which matches the program itself. If I knew the class name for that pop-up, I think I could do the addition. I also have a batch file to launch this program plus the GetData.py. That is invoked with a Windows scheduled task at 4:30 AM on my computer. I expect to post that if there is interest. I will be doing further testing.
    # PopDown1.py
    #import os, sys, glob, time
    import time
    import win32gui as w32
    from win32con import *

    if __name__=="__main__":

    STR_MONEY_IMPORT = "Import a file"
    INITIAL_WAIT = 30
    INTERVAL_DELAY = 2
    IDLE_TIMEOUT = 90

    LogString="" #we will append character for each window find...

    print("Waiting %d seconds to start" % INITIAL_WAIT);
    time.sleep(INITIAL_WAIT)
    IdleTime = 0 # will count up
    while IdleTime < IDLE_TIMEOUT:

    hwndMoneyDlg=w32.FindWindow(None, "Import a file") #normal import
    if (hwndMoneyDlg != 0):
    print ("Did find normal import window.")

    IdleTime = 0 #reset idle timeout
    LogString+= '+' #log a successful find
    w32.PostMessage(hwndMoneyDlg, WM_COMMAND, 2, 0); #Push OK button
    else:
    print ("%2d seconds without new pop-up found." % IdleTime)
    IdleTime += INTERVAL_DELAY

    time.sleep(INTERVAL_DELAY)
    print ("Log string = "+ LogString) #will go to log file...
    localtime = time.asctime( time.localtime(time.time()) )
    logfile= open("PopDown1.log",'a') # open in current directory
    logfile.write( "PopDown1: "+localtime+ "; "+ LogString +"\n")
    logfile.close
    #Cal Learner

    ReplyDelete
  82. I forgot the leading spaces, which are critical in Python, get lost. I made some tweaks, and put the files into PS_day1.zip file at http://www.mediafire.com/?iavip4e6kgrbh1k

    Note PS_Day1.txt and PS_Day1.rtf are the same, except for the format.

    Cal Learner

    ReplyDelete
  83. PopDown2.py available at
    http://www.mediafire.com/?78toaahdslbn26n

    Pops BOTH download error and import complete error pop-up windows. Logs to PopDown2.log in current directory.

    "+++X+" string in log says 3 normal, one error and one more normal pop-ups were detected in that order.

    Cal Learner

    ReplyDelete
  84. Oops again... New link at http://www.mediafire.com/?7i1ax8fd7iy7y7v

    Corrected version has this first line:
    # PopDown2.py 2011/11/11 rev2 Cal Learner

    Cal Learner

    ReplyDelete
  85. Cal: You may want to add a link to some info about the PopDown2.py program. I don't recall it being discussed here (?). It looks like a routine to handle the import dialog in Money (?).

    ReplyDelete
  86. Anyone else suddenly loose Chase downlinks? Looks like the server doesn't even exist any more!
    Old server was ofx.chase.com A DNS look up failed, so it a appears the server has disappeared.

    ReplyDelete
  87. Never mind! Upon looking at old emails, I realized I overlooked one where they told me the service would be down.... Strange that the server has disappeared altogether from the DNS, but hopefully by later today it will be back.

    ReplyDelete
  88. Carlos, I also lost Chase downloads. Do you think they "improved" the service by deleting it? I would love to have them back!

    Harold

    ReplyDelete
  89. Purpose of PopDown2 is to push the OK buttons for you. Can be used for manual or automatic running of GetData.py.

    PS_day1.zip file at
    http://www.mediafire.com/?iavip4e6kgrbh1k (no sign in needed) has explanation and example batch file, but it only uses PopDown1.py -- a version that handles only the normal pop-up. PopDown2 handles the normal and occasional error pop-ups.

    Longer explanation disappeared overnight.

    Cal Learner.

    ReplyDelete
  90. Cal,

    Thanks a lot for PopDown2! It works like a charm and makes downloading faster!

    Harold

    ReplyDelete
  91. Cal,

    Thanks for PopDown2! I incorporated it into my webpage addon for PocketSense rather than using bat file and it works great behind the scenes.

    Dan (ameridan)

    ReplyDelete
  92. I am glad it is useful. One flaw is that if Money is not open, it just keeps thinking that there is a continual download error. Thus the log file for that date would build with an 'X' every two seconds until manual intervention shut it down. I did that when I manually accidentally started a batch file on my laptop before starting Money. I may implement a timeout after maybe 20 successive apparent errors to accommodate the unattended case. Or I could specifically check to see if Money is open; and then print a message, log the fact and return(1) if Money is not already open. That might be a PopDown3.py version.

    The invoking script (program or batch file) could check to see if Money is already open and display a message if it is not. Alternatively the invoking script could check and open Money if it is not already open.

    Another thought that if dealing with the pop-ups were brought into the PocketSense program, it could pace the Money imports (mnyimprt). Upon an upload fail, that particular OFX file could be re-sent.

    Cal Learner

    ReplyDelete
  93. Cal: I haven't had time recently to look closely at the popdown code, but my first thought when I saw it was to add it as a user option in the scripts, with a default = "disabled". I would prefer to check that Money is running via a call to the task list, and abort altogether if it isn't, or even auto-start it. If there's no hurry, I can look into it when I have a bit of silence @ home.

    ReplyDelete
  94. Robert,

    I have a continuing problem with one site, Hewitt Associates. The basic symptom is that I often get an error message from Pocketsense telling me that I have an invalid OFX structure and I should check the file. When this happens, the file is empty. If I retry it again, sometimes the same error occurs, and sometimes it works.

    I don't know Python at all, but I bumbled around the net for a while and inserted a "h.set_debuglevel(1)" to try to see what was going on.

    When it fails with that "OFX structure invalid message", the debug information tells me that I got a "400 Bad Request" reply. I'm not sure why that didn't process as an exception, but it doesn't seem to have done so.

    If I keep on retrying it again and again, sometimes I get a 200 response and all is well, and sometimes I get a 400 response along with the "structure invalid" message.

    Since I have absolutely no idea what I am doing here [smiles], do you have a thought as to what I might try to get this handled?

    Harold

    ReplyDelete
  95. Harold: Does the 400 Bad Request error occur with any other sites (if you have others), or do you ever see it in a web browser session? The script sends the same request format every time, so a random failure may point to something else happening on you PC. I'm assuming you run a good virus checker, etc., and that the system is up to date (patched)?

    I'll look at adding a more graceful check for initial connection codes, and reporting it better.

    ReplyDelete
  96. No, I never get the 400 code for other sites (and I have a bunch of them), and I don't see it in my browser. I'm running totally current levels of Win7 and AVG.

    I realize that it's totally weird that it only happens sometimes. Could it be something on the other end (like, for example, that their server is busy and not defined to handle lots of parallel traffic)?

    I was wondering if I should try calling them (which is akin to beating my head against a brick wall), or perhaps try checking the return code in the Python code, and if it's a 400, retry it in the code for "n" times (which is what I do manually and it almost always works eventually).

    Am I right that I could get a 400 return code, but the current Pocketsense code doesn't check for that, and only sees a problem when it tries to parse the received file?

    Harold

    ReplyDelete
  97. The PS code is currently written to capture errors that can't otherwise be handled (i.e., no response at all from the server, timeouts, etc). I don't currently have anything to check for specific server-side errors. Maybe we can add a generic option that says * retry x times if specific server error conditions occur @ connection*, but default that number to zero? That would keep it from happening when a connection never occurs (e.g., server offline, network down, etc.). (?)

    ReplyDelete
  98. Cal,

    A quick question on your Popdown2 routine that will probably expose my complete and total ignorance of Python [smiles]:

    I notice that for all the Pocketsense routines, there seems to be a .pyc file that matches the .py file that contains the Python code. By looking at the file timestamps, I deduce (?) that the .pyc file contains a compiled version of the .py file.

    My question: How come there is no .pyc file for Popdown2?

    Harold

    ReplyDelete
  99. I searched for this in Google:

    python "no pyc"

    It seems to be because no module imports from PopDown2.py.

    Cal Learner

    ReplyDelete
  100. I replied incorrectly last night, so I deleted my comment above. Try again...

    The purpose for .pyc files is to speed up the loading of imported modules. Importing "test.py" from "my.py" would create a test.pyc module. If test.py is never imported by another, then test.pyc will not be created (unless it is created explicitly by the user).

    ReplyDelete
  101. Robert,

    Yikes. I see that the blogging software deletes leading blanks, which are relevant in Python code.

    I'm hoping that you can follow the code snippet the way it ended up in your blog... if you want me to post again with make-believe "end" statements to delineate the sections of code, I will do so.

    H

    ReplyDelete
  102. YIKES!

    I posted a whole long post about code I added to the Pocketsense scripts to deal with the return code of 400, and it disappeared when I posted again right after it!

    Have any idea why?

    H

    ReplyDelete
  103. [laughs] SOMETHING is going on.

    I essentially rewrote that long post, and posted it.

    And then it disappeared, too!

    How DO posts get deleted here?

    Harold

    ReplyDelete
  104. Harold: No clue about the auto-deletes. It happens to me too! For long posts, I write it in Notepad, and paste it in... to be safe.

    ReplyDelete
  105. Harold: It's probably best to send me code via email, rather than here. pocketsense1 //at// gmail //dot// com. I don't check it every day, nor this site, so I may be slow to respond sometimes.

    ReplyDelete
  106. Cal & Robert,

    Thanks for checking out the .pyc thing... I just wanted to make sure that I wasn't doing anything wrong when I noticed that I had no .pyc file for Popdown2 (which is great, btw).

    Robert,

    I'll recreate my post and send it along via email. Thanks!

    H

    ReplyDelete
  107. Cal or Robert,

    Rather than go to the trouble of autostarting Money as you mention, I think a better method would be to add a simple routine to Popdown (or Getdata prior to password request) that submits an empty ofx file. That would prompt a query to start Money if it wasn't running and do nothing if it was already running.

    ReplyDelete
  108. A batch file can check for *.lrd existing to see if any Money file is open. It could check for abcd.lrd to see if abcd.mny is open. I do that.

    Cal Learner

    ReplyDelete
  109. I know, this is not the appropiate forum, please forgive me
    I have two bank accounts, EmmigrantDirect and BankofInternet that I have not been able to incorporate into PocketSence. But that's ok as long the banks let me use their Money OFX to export into 'Money'.
    Here is my question: All my accounts show up in my favorite accounts and all transactions import just fine, but I can't figure out how to add those 2 bank accounts into my portfolio manager. any idea?
    Gun

    ReplyDelete
  110. answers disappear.

    ReplyDelete
  111. Portfolio Manager is for investment accounts, not Bank (checking) accounts.
    Dan

    ReplyDelete
  112. If you wanted to work around that, you could make the bank accounts to be the cash accounts of fake investment accounts. I would not do that. Instead, I suggest that you look at reports and create favorite reports to show what you are trying to show.

    Cal Learner.

    ReplyDelete
  113. Thanks for taking the time.
    All reports work fine, all totals correct, including bank accounts.
    It's just that I prefer to look at the portfolio manager and I thought it would be nice to have everything includet there. Gun

    ReplyDelete
  114. PopDown2 starts but sits waiting at 20 sec. Then it says waiting nn seconds with not money popup found. What else do I need to do?

    Thanks.

    Mike

    ReplyDelete
  115. Then you need to start Money and the GetData script. All PopDown does is auto-acknowledge the imports and errors for you.

    Dan

    ReplyDelete
  116. Money has an issue with OFX files containing an ampersand (&) in the transaction text. For example, if the name of the Payee is A&B Market, Money will not import the OFX file. If I manually edit the OFX file and remove the "&" and replace it with a "-" or "+" or remove it altogether, the file will import correctly. Can the scrubber script be adapted to remove the ampersand character?
    KR

    ReplyDelete
    Replies
    1. Yes, it is possible. To what extent can you modify code with some help? I guess I should take the fact that you can edit an OFX file as sufficient evidence of ability and motivation.

      I could write a snippet of code that might be worthy of including in general in a later release. It would change any ampersand that is not one of the common allowed sequences, or at least not part of & (I entered the 5-character sequence; lets see how that shows up).

      Cal Learner

      Delete
    2. OK... I wrote it.

      This modification is to replace any ampersand in the OFX file that
      is not part of a proper escape sequence, with the escaped version of the
      ampersand. That sequence is five characters '&amp;' shown with quotes.
      Note that in posting this, I am doing that same thing to make it come out
      right.

      This code is simplified a bit for brevity. To add this mod to your
      scrubber.py file, find line 23, which is
      ofx = f.read() #as-found ofx message

      After that line, add this new line:
      ofx=re.sub(r"&(?!\w+;)", r"&amp;",ofx) # escape singular ampersand

      It is important to have 4 spaces at the start of the new line,
      just as line 23 has 4 leading spaces.

      Cal Learner

      Delete
    3. Cal

      Thank you very much. Your instructions were very easy to follow. The modification appears to be working very smooth. That along with 'popdown' is making Money very usable again. Only have 3 accounts left that I have to manually download.

      KR

      Delete
  117. Hi

    The default website for uploading the quotes is finance.yahoo.com. How can i change that. please help as i want to change the same to in.finance.yahoo.com

    Thanks

    ReplyDelete
  118. India anonymous:
    Edit your sites.dat file to add the following line:
    YahooURL: http://in.finance.yahoo.com

    Cal Learner

    ReplyDelete
  119. Anyone know how I can get Navy Federal Credit Union accounts to download into money? I can't seem to find any informaion on them.

    ReplyDelete
    Replies
    1. They appear to offer web access: https://www.navyfederal.org/account-management/how-do-i.php#history

      If you want to associate the .qfx extension to OFX files so that
      Money can import it directly, you can create a .reg file named
      qif2ofx.reg. Then web access to .qif files would work the same as
      for .ofx files. That .reg file would contain the 15 lines that are
      between the two demarcation lines:

      ==========begin qif2ofx.reg===============================
      REGEDIT4

      [HKEY_CLASSES_ROOT\.qfx]
      "Content Type"="text/qfx"
      @="ofx.Document"

      [HKEY_CLASSES_ROOT\.qfx\ofx.Document]

      [HKEY_CLASSES_ROOT\.qfx\ofx.Document\ShellNew]

      [HKEY_CLASSES_ROOT\MIME\Database\Content Type\application/x-qfx]
      "Extension"=".qfx"

      [HKEY_CLASSES_ROOT\MIME\Database\Content Type\text/qfx]
      "Extension"=".qfx"
      ==========end qif2ofx.reg===============================

      Just run that file after you create it. So thereafter, if you use
      the web etc to bring in a .qif file, it is treated as if it were
      named *.ofx. The Money install associated .ofx with itself.

      If you are also using Quicken, this would probably not be a good
      thing to do.

      Cal Learner

      Delete
    2. Thanks Cal. I guess there's no way to get PocketSense to do this for me since they only offer web connect.

      Delete
    3. I think that's a correct assumption. I just checked in the data files that Money used for bank connections, and it shows Yodlee as the data provider for Navy Federal Credit Union. They typically used Yodlee for banks that didn't offer direct OFX transfers.

      Delete
  120. I made a posting called "Exporting Schedule D Information for US Income Tax to TaxCut or H&R Block At Home" at http://social.microsoft.com/Forums/en-US/money/thread/85fbc12e-76db-4243-b4e6-0c24ddb5557d

    It is not as simple or well-written as I would like, but I think it works. Feel free to comment here or there.

    Cal Learner

    ReplyDelete
  121. I am trying to get the quotes working with Canadian Mutual funds, but so far unsuccessful.
    I have read earlier comments about success and my settings are based on those, but still no luck.
    The script seems to be working for TSX listed stocks like RY.TO, but not for the funds.

    Here is the sites.dat configuration. Am I doing something wrong or Yahoo changed their web site and the YahooScrape does not work anymore?


    YahooURL: http://ca.finance.yahoo.com
    QuoteCurrency: CAD
    EnableYahooScrape: Yes
    YahooTimeZone: -5:EST
    ShowQuoteHTM: No
    AskQuoteHTM: Yes


    F0CAN05MYB.TO s:CA:TDB911 #TD International Index-e
    TDUSINDEXE.TO s:CA:TDB902 #TD U.S. Index-e
    F0CAN05MYF.TO s:CA:TDB909 #TD Canadian Bond Index-e
    F0CAN05NJO.TO s:CA:TDB900 #TD Canadian Index-e


    Please, help.

    ReplyDelete
    Replies
    1. Yes, the format from Yahoo changed. That is the nature of screen scraping. I am working on a patch for you. Maybe later today. Maybe later. Then you should then test it out, and see if it solves the problem for you.

      Cal Learner

      Delete
  122. I am trying to import historical stock and mutual fund (mf) prices by creating an OFX file, using the quotes__.OFX file as a guide. It appears that every stock and mf price appears twice in the quotes___.OFX file created by PocketSense; once in an INVPOSLIST block and again in a SECLIST block. I'll provide more details, if necessary. Does anyone know why there is this duplication? Perhaps I've got something wrong in my sites.dat?

    ReplyDelete
  123. QUOTES4.ZIP is at http://www.mediafire.com/?z7babql3xlufb3q

    It has a replacement quotes.py under a different name, and a txt file. Its purpose is to adjust the screen scraping to changes in how the data is presented by Yahoo server. It worked a few times when I tried it, but back things up first anyway. Report back successes and failures.

    I did not spend a lot of time polishing.

    Regarding the guy making his own OFX files, it's normal. See the specifications at ofx.net

    Cal Learner

    ReplyDelete
    Replies
    1. Hi Cal,

      Got an error: NameError: global name 'day' is not defined

      I think on line 247:
      if qdate > tnow: qdate = datetime(yr-1,month,day)

      Should be
      if qdate > tnow: qdate = datetime(yr-1,month,the_day)

      It seems to be working when set as above. Will test more tomorrow.

      Thanks again,
      Vlad

      Delete
    2. I agree. I uploaded a corrected version at http://www.mediafire.com/?p75rpoaa4rqvvbi
      and deleted the other link. Thanks for the info.

      I dont't know why I did not get the error message in my limited testing, and I don't know why I did not even when I tried again.

      Cal Learner

      Delete
    3. Hi Cal,

      Thanks. The new quotes.py file works, however it gives errors for some quotes like ^DJI, when no month can be found. The error is that variable monthName is referenced before the assignment -> print "the month name is ", monthName #qq

      I added leading spaces after the else: on line 226 until the return csvtxt on line 253. Also removed duplicated lines 229-230 and this fixed the error in case the month is not found. Now the result is the same as the original quotes.py, displaying that Yahoo returned bad data for symbol ^DJI.

      Thanks again,
      Vlad

      Delete
    4. Glad you are making progress, and glad you are getting into the code. Feel free to pretty it up too. That is to say, that I am sure that the style could be improved to make the code less opaque. Screen scraping code with regular expressions is not going to be easy to follow, but don't think that I think that my "style" is very good. Then post a copy of that to sendspace or other place, and we can try running the same file. Then Robert will probably incorporate the modifications of the screen scraper at some point. I followed your comments except the part about the duplicated lines at 229-230. It will be more easier if I can compare your file using WinMerge.

      Lines with 'qq' in them are lines that should not be important, and that I intended to remove at some point. They were temporarily useful in checking the values as I tested and fumbled. I chose that string as something that did not occur in normal text or programs.

      I get most of my actual quotes from the incoming OFX.

      Cal Learner

      Delete
    5. Cal, I am using your code with the small changes above and it works. Thank you for your help.

      Still I may end up using the hleOfxQuotes java program with the Google portfolio as a source, because for some reason Yahoo's TDUSINDEXE.TO quote is not updated since last Aug 30th.
      I see all other Canadian Mutual funds have a new format at Yahoo F0CAN05****, so something is wrong on the Yahoo end for this mutual fund.

      Vlad

      Delete
    6. OK. I see the problems you describe.

      For ^DJI or INDU, there is not a month and day as you said. What I am thinking is that if I find a price but no date to do one or more of the following -- each fancier than the last:

      0. Put a bogus date that will be rejected by later code
      1. Use today's date from the local timezone.
      2. If the time in my timezone is midnight to 8:30 AM, use yesterday's date.
      3. Use today's date unless it is Saturday or Sunday. In those cases, use the date of the previous Friday.

      What do you think? I guess it would be good to see what the time field contains during the weekend.
      Cal Learner

      Delete
    7. Latest version of quotes.zip at http://www.mediafire.com/?mufpt1dxodr2u5r

      I found that ^DJI and INDU Yahoo quotes sometimes have a date field and sometimes not -- probably dependent on time of day. I think that explains why Vlad got different results than I had gotten. This version just makes the other software reject the no-date quote as Vlad's modification does. Later it might be useful to use the date of now or the date of the latest market close when no date is supplied. However just rejecting the quote may be best. That way, the quotes could be fetched later, and if the date is available then the data would be supplied.
      Cal

      Delete
    8. Thanks for the Y! "screen scrape" update Cal! I haven't looked at it (figured I'd wait to see if you needed more updates). Regarding missing dates, my vote is to skip any quote that is missing data. If the data is malformed, then I don't trust the parts received. This seems to be happening with the DJ index, which isn't a "fund" anyways (i.e., it can't be owned in an account). If someone wishes to track indexes inside Money (?), another way may be to track a DJ index fund. It isn't "perfect", but it should be close.

      Delete
    9. Cal,

      I just tried your version of quotes.py and setup.py won't run, so I've reverted back to Robert's. I run the windows version of Python and I infer that you run the DOS version as you state "3. Run setup or Getdata as usual. To test with setup, enter 6 6 Y each time."

      I'm not sure what you mean by this statement, but in Windows the screen just disappears.
      -Dan

      Delete
  124. Cal,

    I just figured out what you mean by "To test with setup, enter 6 6 Y each time." I've always used option 7 and selected Quotes. Anyways, Setup.py doesn't get along with your Quotes.py using the Windows version of Python.

    -Dan

    ReplyDelete
    Replies
    1. When I enter "python" into a command window, I get this:
      "ActivePython 2.6.5.14 (ActiveState Software Inc.) based on
      Python 2.6.5 (r265:79063, Jul 4 2010, 21:05:58) [MSC v.1500 32 bit (Intel)] on win32"

      I am using 32-bit Windows 7.

      Note that you can set up a command window to allow copying to the clipboard if there is an error or debug message that you might want to paste into a post. Do that by clicking the icon at the top left of the command window, and choosing Properties. On the Options tab, tick both Edit options. While there, increase buffer size to let you look back farther. Then to copy something from the command window, select a block of text by dragging while holding down the mouse left button, release, and right-click.


      Cal Learner

      Delete
  125. Hi, is there a way to update the upcoming bill amount in the bills summary using OFX?

    ReplyDelete
  126. By the end of this month, you'll want to insure you are no longer using QWIN appver 1800. I think most everyone should be using appver 1900 (which is Quicken 2011) - the default value that is specified in "control2.py", and in fact I've switched mine to 2100 (Quicken 2012) today, prompted by errors I've been getting on Discover OFX requests the past few days. It turns out that had no effect, but by April 30, 2013 that change will be required anyways.

    Is anyone else getting the message "invalid OFX message structure" from Discover? It takes about 3 tries now before I get a successful data return. Hopefully it is just a temporary issue.
    -Dan

    ReplyDelete
  127. Hi Dan. I just checked, and received the same error. The error being returned from Discover is a generic "Internal Server Error". It was working fine yesterday, so hopefully it's just a server glitch. I'll wait a day or two before getting too curious. Nothing in the error indicates a malformed request.

    ReplyDelete
  128. Hi, has anyone experienced the following error?

    >ERRORPlease verify your identity at http://www.ameriprise.com/quicken to proceed for downloads.<

    Until May 7th this bank worked fine. It seems now you have to go to verify your idnentity and then dowload to Quicken within 30 minutes. I tried my account on Mint.com and it works fine. Any help would be greatly appreciated.

    ReplyDelete
  129. I don't use Ameriprise, but I would think you would go to http://www.ameriprise.com/quicken , enter your username and password. Then in PocketSense, run your GetData.py. Did you try that?

    Cal Learner

    ReplyDelete
    Replies
    1. Yes, that is the first thing I tried. When I do that I get this error: >ERRORAmeriprise Financial is unable to fulfill your request at this time. Please try again later.<. I have Moneydance and they are having similiar issues, so I opened a trouble ticket with them.

      -Mark

      Delete
    2. I don't use Ameriprise, but would be curious to know what the Moneydance community figures out. If they're passing a security token to Quicken, then that would be proprietary (and encrypted). If they're simply validating the host IP, like many banks do now, then it should work with the scripts and Moneydance. Maybe they're looking for a specific Quicken appID value? If it's a newly added (proprietary) handshake, then it violates the OFX standard, and there's not much we can do.

      Delete
  130. I'm using Pocket Sense to download stock and index quotes to Microsoft Money Sunset edition, and it works great. I'd like to include the Dow Jones Industrials Index, but none of the many tickers I've tried works. If you have it, please post. Thanks.

    ReplyDelete
  131. The primary fetch for the quote in csv format fails, so the quotes.py tries the screen scraping function. That fails also, because the format differs from the pattern the screen scrape functions looks for.

    I am looking at making a modified screen scraping function, but I want to make sure that the market-closed format is compatible. I did my initial testing on this problem during market-open time.

    Cal Learner

    ReplyDelete
  132. Has anyone else had trouble downloading transactions from Schwab Bank lately? For about the last two weeks, I've been getting error code 2000 (General error) when I try to download. Otherwise, this same account was downloading fine since February.
    <BANKMSGSRSV1>
    <STMTTRNRS>
    <STATUS>
    <CODE>2000
    <SEVERITY>ERROR
    <MESSAGE>General error
    </STATUS>
    </STMTTRNRS>
    </BANKMSGSRSV1>
    My sites.dat file is set up like specified here: http://www.ofxhome.com/ofxforum/viewtopic.php?id=47480

    ReplyDelete
    Replies
    1. Update: I spoke with Schwab technical support this afternoon. They indicated that they recently changed from an OFX server to a .NET server and had several people call with problems since then. For Quicken users, removing the dashes from the account number resolved the problem. I tried the same thing with PocketSense and now I'm back in business.

      Delete
  133. Back in April of 2011 I had posted about issues with 401(k) downloads. The symptom was that the OFX file had all the individual buy transactions, but Money was ignoring them. What was strange is that this had previously worked (both when Money was downloading directly and through Pocketsense). This happened with both T.Rowe Price and Fidelity 401(k)s. I had not had time to debug, and had resorted to entering most transactions manually. Recently this had become a pain, as I had payments to a 401(k) loan an thus with each contribution there were 3 transactions (contribution, loan interest, loan principal). This finally convinced me to go back and look at the problem.

    Fortunately, just before Money stopped downloading, I had saved all of the OFX log files it had generated (I forget how I did it). Fundamentally, these are just the raw OFX file, but with an extension of text. I renamed the file and tried entering it into a virgin Money file - and it worked! Using the very nice "OFX formatter" utility someone (Cal?) had pointed out I reformatted the Money log files and the latest OFX downloads. Obviously, dates, IDs, and amounts were different, but there was enough similarity to quickly recognize any issues.

    And indeed there was an issue!! Each investment transaction has three fields related to the amount: and Now one would think that Total=Units*UnitPrice. ***HOWEVER*** much to my surprise in the old OFX logs, the TOTAL field was NEGATIVE. I suppose it could be because it is a "negative" flow out of a cash account, although dividend reinvestment show the same negative sign. With this in hand, I took a recent downloaded OFX and did a global replace of with - (Note the minus sign. Bingo! Money processed the file and all its transactions. Now, I just need to write a scrubber of some sort to do this automatically (helped appreciated).

    On a related note, since I had 8+ months to catch up on, I downloaded 8 months worth of transactions (roughly 350+ transactions). BAD IDEA. Money ingested the file and crashed. From that point on, every time I opened the file, Money crashed. Tried a backup - Money crashed. Deleted the actual file - Money crashed. Tried opening sample.mny - Money crashed. Rebooted - Money crashed. Finally had to uninstall and reinstall Money and that solved the problem.

    Hopefully my trials and tribulations will be of helped to someone.

    ReplyDelete
    Replies
    1. Good to hear Carlos, but I'm a little confused. Looking at investment downloads from Vanguard, the TOTAL field is always negative for buy transactions (amount and price are positive), but I don't know how sale transactions appear (I don't currently have any to check). It sounds like T. Rowe Price and Fidelity are doing something differently w/ their OFX downloads. Is this peculiar to 401K accounts only, or all investment accounts with them? I'm assuming that the sites.dat entry shows a INVSTMT type account for the 401k, but do you have any non-401k accounts at the same brokers that work?

      Thanks for sharing your effort.

      Delete
  134. Robert,
    I will have to look some more to see I discern any patterns. I do have both accounts as INVSTMT. Unfortunately I don't have any other accounts at T.Rowe Price, and Fidelity 401(k)s are handled by "Netbenefits.com" rather than "fidelity.com" so it is a different server. I think something changed on their end. Upon further inspection I found one other strange thing. It used to be that when I downloaded from TRP I would get two sets of transactions: the deposits into the cash account, and then the buy transactions to the investment account. I have confirmed that this was the case when MS$ was still downloading (again since I have the old log files). I _think_ that it was also the case when the pocketsense downloads where still working, based on which transactions show up as "E"s in money. The new downloads though, don't have the cash account deposits, and have the positive field. I don't really care about the cash deposits (since those are in my automatic bills, and always the same amount). All I care is about the buy transactions - which is why I'm thinking of just writing a quick scrubber.

    ReplyDelete
  135. This comment has been removed by the author.

    ReplyDelete
  136. Do you have any plans to add support for CLIENTUID into the scripts. One of my banks require it, along with OFX version 103.

    I have made a patch like fix that is working for me. It would be nice to not have it hard coded in the ofx.py file.

    As far as I can tell, this is simply an authentication step for some banks, where it uses the Client UID, to identify that the same client is requesting data every time. So an option to define it in the sites.dat would be nice.

    James

    ReplyDelete
    Replies
    1. I really haven't looked at it before, but I did a quick search to learn a little more. It appears the the ClientUID parameter is indeed another level of security, and basically acts as another password to identify the client instance. What isn't clear, is whether this value is provided by the bank or is self-created and registered by the client. According to some discussions, the client should be able to create this value, but I'm not sure how that would provide any extra user/account security. If the server (bank) provides the value, then it makes sense. It also appears that this parameter requires the OFX version id to be 103 (as you mentioned), or more accuratly... that an OFX connection using version 103 requires the ClientUID parameter.

      Since I don't use a ClientUID for any of my accounts, I would need to know how it's actually being used. Is the bank providing the value as a static parameter, or is it being "made up" and handed to the server?

      Delete
    2. I am using a static entry for it from the client. I handled it by adding a configuration field to site.dat. Then if that field is defined, it changes the header to 103 and supplies the provided CLIENTUID to the BANK. From what I can tell, it is generated by the client and provided to the bank. If you would like, I can provide my code to you.

      I am using PocketSense with a budget program called YNAB. I have made some changes to the program to make it work more smoothly with YNAB. So my code actually has quite a few changes to it.

      However, here is the overview of my changes.

      There is also a bug in the code. If you are downloading multiple accounts from the same bank, and the download goes fast enough, the program may generate the same file name for both files. I fixed it by putting the last 5 characters of the account number in the file name.

      Added a question in getdata.py to ask for the number of days to retrieve. It defaults to default number of days in site.dat

      added code to combine all of the OFX files before sending to YNAB. This improves YNABs handling of all of the imports.

      James

      Delete
    3. Also, Thank you for all of your work on this. PocketSense is a great tool for automating the download of OFX.

      Delete
    4. Thanks James. I'd like to stick w/ the CLIENTUID issue to begin with, so I'd like to clarify. It sounds like you "made up" the value, and pass it to the bank during connection (?). I've read that some banks provide a hex number, but it doesn't sound like you needed to do that. Did the value need to have any particular format (hex? length? bit pattern?). If you'd rather, you can send messages directly to pocketsense1 /at/ gmail /dot/ com. I've been slow to keep up w/ this for a good while now, as I've been (and continue to be) swamped in my job and family life, but this (plus a few other odds and ends) will be worthwhile updates I think.

      Delete
  137. Has anyone got Bank of America to work? I have tried all 4 of the FID/URL links, and on setup and testing the connection I get one of two error messages. If anyone has successfully downloaded BOA bank statements, please reply to this.

    Error mesaage 1
    Code:2000
    SEVERITY:ERROR
    MESSAGE:The product which you are using is not enabled. Please call customer service at 1.800.933.6262 for assistance.
    OR
    Error message 2
    MESSAGE:Bank of America has completed a planned systems update for accounts opened in California. Please visit www.bankofamerica.com/Financialmanagement for detailed instructions on how to update your settings and avoid service interruptions.

    Thanks
    Uday

    ReplyDelete
    Replies
    1. I've been using BoA for years with Money, and for almost two years with the Pocketsense scripts. So the good news is that it works. The bad news is I don't recall if I ever "enabled the product" since it has been so long. I have a Maryland based account with both checking and credit card. I use the same log-in as I do for the web site - although obviously without the site challenge.

      Here's the portion of the entry from my sites.dat file (both credit card and checking are identical, except that credit card has a blank "bankid"):

      fiorg : HAN
      fid : 6812
      url : https://ofx.bankofamerica.com/cgi-forte/fortecgi?servicename=ofx_2-3&pagename=ofx
      bankid : 052001633
      brokerid :
      appid : Money
      appver : 1600


      Delete
  138. I don't know if folks keep up here much, but I've recently made a few changes to the scripts. The updates are in response to various requests over the past year, but James lit the candle when he asked about support for OFX version 103. The changes he had made for himself gave me the urge to revisit some work I had done last year regarding combining OFX files prior to sending to Money. It wasn't as straightforward as I originally hoped, so I had dropped it. After some poking and prodding, I think I have a working method now. I have posted a beta version with the following changes, should anyone want to test it.

    - Support for OFX 103 and ClientUID parameters. The ClientUID is defined by site entry in sites.dat, and the ClientUID value is automatically created, and saved back into sites.dat as a static UID value.
    - New sites.dat option named "promptInterval". If defined to "Yes", the user is prompted for the download interval (days) as an override to the global default and site minimum values defined in sites.dat. (default=No)
    - New sites.dat option named CombineOFX. If defined as "Yes", ofx statement files are combined into one file prior to sending to Money. Default=No. I have tested this with my own accounts, but it took a bit of experimenting to find the right way/sequence of combining things.

    The beta (test) version can be downloaded at https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip. The sites.template file has placeholders and instructions for the new parameters.

    I still need to revisit some notes that Cal sent previously regarding replacing with in statements, and a special adjustment for Fidelity dates (iirc).

    ReplyDelete
  139. Thanks so much for posting Robert. I was going to ask you to incorporate the popdown2 script, but it won't be necessary if the ofx files are combined. My first run of the beta scripts seemed fine, but since I had already run the official scripts today, there were no additions, so I'll let you know tomorrow. Also, could you take the error part of the response to the download interval prompt out? I plan on just hitting return most of the time (to accept the default) rather than inputting "7".

    -Dan

    ReplyDelete
    Replies
    1. New Beta scripts running fine here. Don't know about the 103 files since I do not have any. I did my usual incorporating of the "popdown" with a flag in sites.dat and it's working also.

      Ray

      Delete
    2. Oops... the error message wasn't a "feature", it was a bug. Thanks Dan! I have the prompt option disabled on mine, so I must have changed something between first testing and posting it. Beta updated.

      Delete
  140. Hi all - running Windows 8 and there is a oft reported Money crash when running win 8 and money when downloading data. I tried running Money in a Virtual Machine of Win 7 but now can't access using Pocketsense as I'm getting Socket.Timeout errors.
    Any ideas how to solve?

    ReplyDelete
  141. I can't believe how much smoother the combined OFX file is processed vs. individually. Like the old days when Microsoft servers handled the downloads. The revised beta is working well for me.

    -Dan

    ReplyDelete
    Replies
    1. Loaded in the latest Beta. Yes the combined OFX is much faster, unfortunately more than half of the accounts are missing.

      I have five accounts: 2 - Credit Cards, 2 - Bank Accounts, one 401K. The combined file had the 401K and one credit card, that's it. I do have downloads for all five accounts.

      I noticed the same omissions on the Beta from Aug 23 and the latest from Aug 27.

      Ray

      Delete
    2. That's not good. Are you saying that the combined ofx file is missing transactions that do exist in the individual account files? The "combiner" routine will not import null ofx files (i.e., files that don't have statement sections in them). If you open the individual account file, does it import anything? Let me know, and I'll dig a little deeper tonight.

      Delete
    3. Just finished doing some more digging.

      This is what I found.
      If the original downloaded file from the bank or credit card company has cr/lf ( 0x0D 0x0A ) characters between the "OFX /OFX tags", then the file is ignored. If it is straight data (ie no line terminators), then the file is included in the combined OFX.

      That is the only difference that I can see right now.

      Ray

      Delete
    4. Definite bug. Fixed (I think). Beta updated.

      Delete
  142. I forgot to mention a new parameter in sites.dat for the last upload (w/ the ofx cr/lf bug fix). The name is quietSrub, and if "Yes", will suppress the scrubber messages. I don't see a reason to suppress them myself, but there's probably not much reason to see them either. Default=No.

    ReplyDelete
  143. Not sure if I messed something up, but if I use the newest Beta I get the following error:


    Traceback (most recent call last):
    File "C:\DOS\OFX_Beta\Getdata.py", line 118, in
    cfile=combineOfx(ofxList) #create combined file
    File "C:\DOS\OFX_Beta\rlib1.py", line 280, in combineOfx
    bantrn, crdtrn, invtrn, sectrn, ''])
    TypeError: sequence item 3: expected string, tuple found


    If I replace rlib1.py with the previous version, There is NO error ( of course the file is bad with the OFX cr/lf bug, but it does not crash.

    Ray

    ReplyDelete
  144. FYI Robert

    I just downloaded and reloaded the scripts keeping only my sites.dat and ofx_config.cfg files. I then ran Setup and Getdata. The files downloaded fine.

    Then I changed sites.dat to enable combineofx, ran Setup and Getdata again. This time I got the same error listed in a previous message.

    Ray

    ReplyDelete
    Replies
    1. Do you think replacing
      ofx = ofx.replace("\n","")
      with
      ofx=ofx.rstrip()

      would work? It does not crash. No more time to test tonight.

      Ray

      Delete
    2. Thanks Ray. Beta updated... try again. I can only imagine that some files aren't using the exact cr/lf sequence. Unix/Linux/etc doesn't necessarily use cr/lf (may only use LF), so that's a guess. The regex requires a flattened input string w/ no terminators. If this doesn't work, then there's something different that we need to figure out. BTW, I have one account that uses cr/lf that didn't work in the first version, but worked on the last and this one. I missed it at first, since it doesn't have any recent transactions.

      If anyone is using a bank account, can you verify that BANKMSGSRSV1 is used to start a statement? I don't have any, so I used what I found from a search. Need to verify...

      Delete
  145. Robert,

    I have a bank checking account and the lines following the SIGNONMSGSRSV1 section is BANKMSGSRSV1 and then STMTTRNRS.

    -Dan

    ReplyDelete
  146. Just ran the latest beta using the combine option, and it craps out, so I'll have to await the next revision I guess.

    ReplyDelete
  147. Running the new Beta I get the following error:

    Traceback (most recent call last):
    File "C:\DOS\OFX_Beta\Getdata.py", line 118, in
    cfile=combineOfx(ofxList) #create combined file
    File "C:\DOS\OFX_Beta\rlib1.py", line 299, in combineOfx
    bantrn, crdtrn, invtrn, sectrn, ''])
    TypeError: sequence item 3: expected string, tuple found


    If I replace the lines:
    ofx = ofx.replace(chr(13),"") #remove CRs
    ofx = ofx.replace(chr(10),"") #remove LFs



    with:

    str1=""
    idxofx = ofx.find(str1)
    str2=ofx[0:idxofx]
    str3=ofx[idxofx:len(ofx)]
    str3 = str3.replace(chr(13),"") #remove CRs
    str3 = str3.replace(chr(10),"") #remove LFs
    ofx=str2+str3

    I get the same error:

    Traceback (most recent call last):
    File "C:\DOS\OFX_Beta\Getdata.py", line 118, in
    cfile=combineOfx(ofxList) #create combined file
    File "C:\DOS\OFX_Beta\rlib1.py", line 299, in combineOfx
    bantrn, crdtrn, invtrn, sectrn, ''])
    TypeError: sequence item 3: expected string, tuple found


    Looks like removing the cr/lf includes a file in the combineofx, but must be uncovering another error.

    Hope all this makes sense to you.

    Ray

    ReplyDelete
  148. I had that error while editing another part of the combine function this morning, but thought I'd fixed it. The error shows line 299, but the most recent version of rlib1 only has 288 lines. Hmmm (?)... it's as if my previous upload didn't replace the bad one (or I didn't do it right). Uploaded again, and am really curious.

    ReplyDelete
  149. Sorry about the line numbers. I added a few lines of trouble shooting info, just go by the line contents.

    Ray

    ReplyDelete
  150. With the latest rlib1.py date 8/29/12 6:01 PM


    Traceback (most recent call last):
    File "C:\DOS\OFX_Beta\GetdataX.py", line 132, in
    cfile=combineOfx(ofxList) #create combined file
    File "C:\DOS\OFX_Beta\rlib1.py", line 281, in combineOfx
    bantrn, crdtrn, invtrn, sectrn, ''])
    TypeError: sequence item 3: expected string, tuple found

    Something about this statement:

    combOfx = '\n'.join([OfxSGMLHeader(),
    '', signon,
    bantrn, crdtrn, invtrn, sectrn, ''])


    Ray

    ReplyDelete
  151. FINALLY I think I found it.

    The line:
    if bantrn: bantrn = OfxTag('BANKMSGSRSV1\n', bantrn)

    has a comma at the end. Removed it and the file created without error. You still have to remove cr/lf.

    I will test further to see if it actually works.

    Ray

    ReplyDelete
    Replies
    1. I think you found it Ray. I don't have a bank statement download, so mine would have skipped that line (i.e., bantrn is always null for me). Beta updated... again. Thanks!

      Delete
  152. That's good that your bank has an OFX server for you. I have to use the web to download an OFX for the bank. I will try to find that if I switch banks at some point. Credit cards and brokers with OFX servers are more common.
    Cal Learner.

    ReplyDelete
  153. Good eye Ray.

    I tried to find something out of place with no success. Yes, the beta works great now!! All of my accounts were in the combined OFX file, including my bank (PNC).

    -Dan

    ReplyDelete
  154. In the latest beta, I believe you need to change ofx.py on lines 237-238 from

    if minInterval:
    interval = max(interval, interval) #use the longer of the two

    to:

    if minInterval:
    interval = max(minInterval, interval) #use the longer of the two

    The code is ignoring minInterval as defined in sites.dat. Sorry about the formatting, it looks like leading whitespace is stripped from each line.

    ReplyDelete
  155. For the only bank I have (multiple accounts), I keep getting an "Invalid OFX message structure" error and the ofx file is blank. Here's the infor from the sites.dat file:


    SiteName : CPFCU
    AcctType : BASTMT
    fiorg : Cedar Point Federal Credit Union
    fid : 255077736
    url : https://pcu.cpfcu.com/scripts/isaofx.dll
    bankid : 255077736
    brokerid :
    ofxVer :
    appid :
    appver :
    mininterval:

    ReplyDelete
    Replies
    1. The "invalid" message indicates an incomplete OFX response from the server. Assuming that these are bank accounts (checking/savings), then the other thing to check is your entries in setup.py. Verify the account number formats, usernames, passwords, etc. A common problem folks have had is to omit leading zeros on accounts, or using the wrong case passwords. I would recommend deleting the accounts, adding one at a time, and testing that one account before adding more.

      Delete
    2. Thanks for your reply. I will delete the accounts and add them back. I haven't fully looked though the code, but this bank worked with the previous (non-beta) version. I just copied the ofx_config file from the old working version to the new version.

      Delete
    3. If that's the case, then deleting and recreating shouldn't be the answer. I don't believe anything was changed that would alter the original OFX file download. Did you enable any particular option in the beta, or are you running it the same as before (i.e., new options disabled)?

      Delete
    4. I enabled the CombineOFX option. Other than that, everything remained the same. It downloads my credit card statements from Amex, Citi, Chase, & Discover just fine.

      Delete
    5. Run setup.py and test each account individually (option 7). If it fails there, re-install the non-beta version and try again.

      Delete
    6. One step ahead of you :-). I had already done the test for each account and it failed. I have the non-beta version in a different directory and tried that and it passes just fine. I'm going through the getOFX function in ofx.py to figure out the differences between the two versions. I'm not sure how much I will be able to decifer.

      Delete