Friday, March 24, 2017

New version released

(comments closed)

Updated:  10-Apr-2017 

Spurred by recent OFX server changes at Discover (and apparently others), the script package has been updated, and is posted for download @ https://sites.google.com/site/pocketsense/home/msmoneyfixp1/p2.  Thanks to everyone for feedback and help!  I think we're in good shape for a while, but it would be really nice if the bank developers would abide by the OFX standard ;)

Summary of Changes

  • Updated url requests to work with recent changes at Discover and others.
  • Added support for OFX 2.x (tested with ofxVer=211 @ Discover, but they support 102 and 103 too).
  • Added support for user-specific clientUID pairs (by site url + username).
  • Added skipZeroTransactions option to sites.dat configuration.  If Yes, remove $0.00 transactions from downloaded statements.
  • Minor code updates and bug patches.

70 comments:

  1. Hi Robert. With the end of the month I was finally able to test the sign fix for Net Benefits in the new version. It turns out it doesn't work, but I've spotted the reason why.

    INVBUY and INVSELL transactions have the following order

    REINVEST has a different order (of course, why not, why have a standard?):

    As a result the regex that's used in _scrubINVsign does not pick up REINVEST transactions

    Thanks
    Carlos

    ReplyDelete
    Replies
    1. Thanks Carlos. I think something's missing from the post above?

      Delete
    2. Ugh. I forgot tags get interpreted by the comment engine.
      INVBUY and INVSELL transactions have the following order: UNITS, UNITPRICE, TOTAL
      REINVEST order is: TOTAL, UNITS, UNITPRICE

      I can provide you with an example, if you want. It's a trivial change, but a different scrubber is needed for REINVEST than for INVBUY/INVSELL.

      Delete
    3. Gotcha. Pretty sure I know what's needed, but since I can't test, it would be quickest if you send me the scrub routine you're using for REINVEST transactions, and I'll add it. Send to pocketsense1/at/gmail/dot/com.

      Delete
  2. Thank you so much for your work on this Robert!

    -ameridan

    ReplyDelete
  3. New version posted, including scrubber update provided by Carlos. Specifically, malformed REINVEST transactions should now be handled correctly.

    ReplyDelete
  4. Um, _scrubREINVESTsign is not actually called anywhere, is it?

    ReplyDelete
  5. Yeah Robert, thanks for maintaining this site. This has been a great tool to keep MSMoney alive.
    So, I was wondering if it would be a challenge to add a feature in SETUP to re-arrange the Site list order of execution. Thanks again!

    ReplyDelete
    Replies
    1. I've been tied up on other challenges, so a bit behind here. When you say "rearrange", do you mean a sort order, or a "user decided" order? I'm also wondering "why"?

      Delete
  6. I'm trying to set up MSMoney and I've come accross a couple of problems. The first is when I try to run the Setup.py file ,nothing happens. I'm running Windows 10 - 64bit. The second is the link (http://www.ofxhome.com/index.php/home/directory) times out and the saved copy of this site does not allow to look in the directory, using the other option (http://moneydance.com/synch/moneydance/fi2004.dict) I can only find banks that are based in the U.S.

    ReplyDelete
    Replies
    1. Not sure about ofxhome, etc., but curious if you got the scripts working? If nothing happens via Setup.py, then that's likely a problem w/ the Python install. Generally, ActiveState Python configures everything correctly in Windows, but some do not.

      Delete
  7. Found this while looking for a fix to my problem (https://pypi.python.org/pypi/csb43/0.3.2),
    Tools for converting from the Spanish banks’ format CSB norm 43 (CSB43),
    (norma 43 del Consejo Superior Bancario [CSB43] / Asociación Española de Banca [AEB43]),
    Supported formats:
    OFX XML
    HomeBank CSV
    HTML (python2)
    JSON
    ODS: OpenDocument spreadsheet (python2)
    CSV, TSV: comma- or tab- separated values (python2)
    XLS: Microsoft Excel spreadsheet (python2)
    XLSX: OOXML spreadsheet (python2)
    YAML
    might come in handy for others.

    ReplyDelete
  8. Anybody else getting Error code 2003 when attempting Discover Card downloads? I'm hoping it's temporary, but 3 days now...

    ReplyDelete
  9. I thought I might be one beta behind, so I just downloaded the latest version of Pocketsense, retried Discover, and now I get "Access Denied; You don't have permission to access "http://ofx.discovercard.com/" on this server."

    ReplyDelete
  10. Yes I'm getting same access denied error with discover again [sigh]. Was working fine with Beta for many weeks, then stopped working so I decided to upgrade to latest release but alas it still didn't work. Another discover problem I suspect.

    ReplyDelete
  11. I tested just now, and have the same result. Will look into it...

    ReplyDelete
  12. I downloaded a recent Discover Card statement (qfx format) using the web portal, to see if anything had changed. The account number is different, having a format of "Discovr60xxxxxxxxx####", where the #### looks like part of the original account, and the xxx... is something unique. I added a new account in Setup using this account# (exactly), with the same user/password, and tested w/ success.

    I repeated the above for Discover Bank, since I use them as well, and found the same type of change. It too worked with the (new) encoded account#.

    Be sure to test during Setup, and send the statement to Money and re-match it to the existing account @ import. Since it has a different acct#, it must be re-matched when prompted.

    I noticed the longer acct# doesn't fit in the existing Setup list output, so I'll widen the account column.

    ReplyDelete
    Replies
    1. Hi Robert - thanks for keeping us Money diehards going!

      When implementing the above acct # setup for Discover Card, transactions download into Money Sunset again, however subsequent downloads reset previously-Reconciled transactions as Unreconciled, making me set as Reconciled again. Does this happen to you as well?

      Delete
    2. (Same issue for Discover Bank...)

      Delete
    3. Derek, I haven't experienced that. On the other hand, transactions downloaded from the web page won't be recognized when repeated using Pocketsense downloads, since the script alters the ID somewhat and perhaps you deleted the wrong repeats?

      Delete
    4. Just tested w/ 3 subsequent downloads via Pocketsense (Discover Bank), and the same items now appear 3 times in Money.

      Delete
  13. Great job Robert! That worked for me too.

    ReplyDelete
  14. Robert you are a genius! Works great again. Mine was in the format DiscovrXXXXXXXXXXX#### where #### was the last 4 digits. I did not have the 60 to start. But that random number worked. Must be a security thing for Discover that they just failed to tell anyone or put on their help pages. Thanks Again!

    ReplyDelete
  15. This is so weird.... I have a few Discover Bank accounts, and each one has a different value for "XXXXXXXXXXX"... I had to download each one using the web site and look at the file!

    ReplyDelete
  16. Updated to the May 11 version, but I'm still getting Discover errors. Any suggestions?
    OFX file shows:

    You don't have permission to access "http://ofx.discovercard.com/" on this server.
    Reference #18.c9ee3e17.1494689640.146a2b8

    ReplyDelete
    Replies
    1. See post above (5/12/17 @ 727a)

      Delete
    2. Hi Robert,
      thanks for the quick response!
      Now I understand... instead of using my discover card account number, I need to download a qfx file from the website and locate the account number in that file.

      Delete
  17. Python question: I copied a *.mny file from Win7 to a different PC with Win10 and ever since the "get data" does not work on the Win10 PC, Pop up says: python.exe stopped working Windows will close the program. Would it help to reinstall Python 2.7 and if so, would the new program install itself onto the old or should I uninstall the old Python first?

    ReplyDelete
  18. So weird. Thanks for discovering the Discover fix Robert.

    Also, I expect this as come up before but does anyone else get a bunch of zero amount transactions when downloading from Discover? I get so many of them (they only have payees but no memos or amounts). It'd be interesting if there's a way to stop those from being included in the ofx file.

    ReplyDelete
    Replies
    1. The newer versions of the script's sites.dat file has a 'skipZeroTransactions' Yes/No setting.

      Delete
  19. Robert,

    Thanks for this great work. Is there any chance that you have source control, or history of your scripts somewhere? I'm also using a script that appears to have been originally derived from the same source, available as the "ofxclient" package on PyPI or at https://github.com/captin411/ofxclient

    I was using it to download from Discover, and it broke last week. I was hoping to find a way to merge your OFX 2.0 work into it.

    Thanks,
    Jason Antman

    ReplyDelete
    Replies
    1. Sorry Jason... but no source control. Probably should have, but never did.

      Delete
    2. I have the following downloaded zips of pocketsense:

      ofxpy_pocketsense_beta-20130217.zip
      ofxpy_pocketsense_beta-20170307.zip
      ofxpy_pocketsense_beta-20170312.zip
      ofxpy_pocketsense_beta-20170315.zip
      ofxpy_pocketsense-20130714.zip
      ofxpy_pocketsense-20140906.zip
      ofxpy_pocketsense-20170324.zip
      ofxpy_pocketsense-20170410.zip

      plus a ofxpy_pocketsense_beta.zip where the latest file (rlib.py) is dated 2012-08-29.

      Each individual archive is about 40kb. I'm sure if I've missed any, someone else will have a copy. Robert may already have all of these.

      From these versions, it should be fairly easy to initiate source control - just start with the oldest and check in the changes in order.

      Delete
  20. Chase and Discover down this morning. Hoping it is temporary. Chase gives empty OFX files (0 bytes). Discover gives ERROR OFX status.

    ReplyDelete
    Replies
    1. I guess I posted too quickly before reading the thread. Downloaded file directly from web site, and like Harold and Robert said, it has a random account number. Changed the account number to the new random number, and all is well. Let's hope they don't start changing it every 5 minutes.

      Delete
  21. Hi Robert,

    I have been a long time MS money user and have not found a alternative that can do everything MS Money does. I found your scripts very useful to update currencies and investments however none of my bank accounts are supported by the OFX servers.

    Do you perhaps know if it is possible to use yodlee or any other source(s) to automatically extract the information and then import into ms money via your scripts? That will allow users like myself (outside US) to gather all their information and import into ms money. Also how much effort programming will be involved.

    All other software alternatives have some sort of limitations, either being functionality or only us supported.

    ReplyDelete
    Replies
    1. Can't really help w/ Yodlee, as I've never used it.

      Delete
  22. Hi, Robert.

    It looks like your scripts are set up to grab the end-of-day close for the day you run GetData.py. Do you have anything like TheFinanceBuff's old histquot.py, where I can run it once a week to grab the weekly close for stocks and mutual funds at my leisure? I'd run "python histquot.py 20170519", for example, to get last week's close data and import it into MS Money. Yahoo shut off their API last week, and I tried reprogramming it for the Google Finance API. I can get it working for stocks (e.g., VTI, AAPL), but not for mutual funds (e.g., MUTF:VDIGX).

    Thank you!

    ReplyDelete
    Replies
    1. Hi Dan: Didn't realize Yahoo had shut off their finance interface. The Google quote interface extracts results from their current-quote page, and I haven't looked at historical quote options.

      Delete
  23. So I've moved my banking to Bank of America (in Texas). Been trying to get pocketsense to pull the data from it (a checking and a savings).

    Have HAN and 5959 for the fiorg/fid; a downloaded file from the website has bankid of 111000025 so been trying that (as well as a couple of others I've seen on the internet). I know my user and pass are correct when I set it up, but I still get an error ofx that says "An incorrect username/password combination has been entered. Please try again.".

    I've read a few places that say I need to pay extra to BoA for Quicken access but I'm not seeing an option on their website to add it. Does anyone have any insight?

    ReplyDelete
    Replies
    1. If I remember correctly, you need to call and tell them you'd like to access your account with Quicken, and they flip a switch. It has been many years, so YMMV

      Delete
  24. Anyone noticing issues at Vanguard? Haven't been able to get ofx lately. Temp ofx file is blank, not giving me a direction to look into. I'll post back if I find anything, but mainly curious if others are having similar issue.

    ReplyDelete
    Replies
    1. Same here, but waiting to see if self heals before focusing on it. It's happened before w/ Vanguard (i.e., extended offline period).

      Delete
    2. Same here. getting invalid ofx with Vanguard. Download from Vanguard site works fine.

      Delete
    3. Yes, noticed that my Vanguard account which has been working fine for months has stopped downloading transactions. .ofx file in the xfr folder is completely empty, no clue what's wrong. Manual download from the website still works. If anyone is able to get it working please post with info.

      Delete
    4. I too am having this problem and I've been using PocketSense with Vanguard for years without any issues until now.

      Delete
    5. I'm not using PocketSense, I'm using the Python ofxclient application, but that appears to be originally derived from this code so a lot of the logic is the same. I download OFX for all of my accounts daily at 2am Eastern, and Vanguard worked up to July 21, 2017. Since then it's been sending the empty responses. I dug around a bit and tried different app and OFX versions, but nothing seems to help...

      Delete
  25. I have been downloading from Vanguard for years with ofx4j and it recently stopped working for me too. This is the only "current" thread I found on the Internet where folks were discussing the issue.

    Oddly, if I run Postman and send the exact same message I do through ofx4j it works and I get a valid OFX response - but Postman won't give me a detailed network trace of what was sent / received. The exact same request via ofx4j responds with an http 200 (Ok) but the body is empty...

    The Firefox plugin HttpRequester which does log network activity doesn't get a response body either :(

    Any suggestions / ideas on what Postman could be doing differently and why it is working when other things don't.

    ReplyDelete
    Replies
    1. Very interesting about Postman. There seem to be a few ways to capture the data... Google turned up https://www.getpostman.com/docs/postman/sending_api_requests/capturing_http_requests and http://blog.getpostman.com/2015/06/13/debugging-postman-requests/

      On the other hand, if you could provide an anonymized version (minus your username, password and account number) of what Postman is sending, I could try and figure out what's needed to get it to work...

      Delete
    2. Using the HTTP generate code snippets feature of Postman, you get the following anonymized request:

      POST /us/OfxDirectConnectServlet HTTP/1.1
      Host: vesnc.vanguard.com
      Accept: application/x-ofx
      Content-Type: application/x-ofx
      Cache-Control: no-cache

      OFXHEADER:100
      DATA:OFXSGML
      VERSION:102
      SECURITY:NONE
      ENCODING:USASCII
      CHARSET:1252
      COMPRESSION:NONE
      OLDFILEUID:NONE
      NEWFILEUID:0e27a5ae-f76e-49ef-96ec-d2044609be2c

      <OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20170720214559.879<USERID>XXXXXXXX<USERPASS>XXXXXXXX<LANGUAGE>ENG<FI><ORG>Vanguard<FID>15103</FI><APPID>QMOFX<APPVER>1900</SONRQ></SIGNONMSGSRQV1><SIGNUPMSGSRQV1><ACCTINFOTRNRQ><TRNUID>09486e1e-53f3-4ec4-940d-8cd05b458895<ACCTINFORQ><DTACCTUP>19700101000000.000</ACCTINFORQ></ACCTINFOTRNRQ></SIGNUPMSGSRQV1></OFX>

      Delete
    3. And I had seen those notes about capturing requests but they use the Chrome Postman extension not the standalone client and the Chrom Postman client doesn't work but the Standalone client does (should have mentioned that originally).

      Delete
    4. I think I found the culprit, but need to test a bit before posting a solution. Vanguard appears to be using (requiring) session cookies, which I figured out by playing around w/ Postman mentioned earlier. Stay tuned... but I don't think it will be hard to remedy.

      Delete
    5. Cookies are definitely the culprit - if I capture cookies in a first request and make a second request passing in the cookies everything works as expected.

      Thanks everyone!

      Delete
  26. None of my Schwab accounts are downloading this weekend. I'm hoping it's because the server is down for maintenance, but I've never seen this error before:

    ** An ERROR occurred retrieving POST response from ofx.schwab.com
    Exception type :
    Exception val : The read operation timed out

    ReplyDelete
    Replies
    1. After calling Tech Support at Schwab to have my IP address added to their "White List", all is well again. I didn't try out the beta version; I'm still using the production version of Pocketsense.

      -ameridan

      Delete
    2. Thank you for the advice. I called Schwab Tech Support and they were very helpful. They said they had been fielding "many" calls about this issue. They put my IP address on this "white list" and everything is working again.

      Phil

      Delete
    3. I'm curious, but does anyone know whether Schwab has implemented an automated 2-factor authentication protocol? I ask, because I'd bet other packages are being blocked too, and user IP addresses are dynamic... meaning they will change, and they'll either be dealing w/ this constantly, or stopping connections altogether.

      Delete
    4. I'm not sure about any new authentication protocol, but the tech said most of his calls had come from Quicken users.

      Delete
    5. Robert,

      Schwab Tech Support claims that they enter a range of addresses (as most people aren’t lucky enough to get a static IP address) and not your actual address at the time of your tech support call.

      The authentication is still the original 1-stage username/password.

      Delete
  27. I get the same "timed out" error with my Schwab Brokerage and Bank accounts.

    ReplyDelete
  28. A new version is available for testing, and can be downloaded @ https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip . The patch is specific to the Vanguard issue (i.e., use of cookies), so I doubt it helps w/ the Schwaab problem. Could be surprised, though? I don't use Schwaab, so I can't test that one.

    I'm not certain that the way I've handled session cookies will always work, so it may need revision at some point.

    ReplyDelete
    Replies
    1. Is there any chance you have a pointer to the current version, or could upload/paste/post the changes you made to fix this, somewhere?

      Delete
    2. The current version is still active on the main Install page @ https://sites.google.com/site/pocketsense/home/msmoneyfixp1 . The above link is a "beta"... for testing.

      Delete
    3. Should add... pretty sure the only changes of significance were to ofx.py and rlib1.py.

      Delete
    4. THANK YOU! Can confirm that this appears to fix the Vanguard issue. I am now once again able to download my Vanguard account info. I will keep a close eye on it in the next few days to make sure everything keeps working.

      Delete
  29. mabrownmab at yahooJuly 31, 2017 at 10:54 PM

    Well, after seeing all the comments, I'm glad I asked the community and I'm really, really grateful to Robert for implementing solutions (and to those that help find the solutions). I can confirm this worked for my Vanguard accounts also.

    ReplyDelete