1. Import manually downloaded files
- Setup now creates an ./import sub-folder (alongside ./xfr, etc). Running setup creates the folder, but you can add manually too.
- Any ofx, qfx, etc. file can be dropped into the folder. No need to rename.
- When Getdata.py is run, it will process *all* files in ./import, and if it "looks like" an ofx file, it will be treated the same as a downloaded file (scrubbed, combined, etc).
- If accepted as a valid ofx file, it is moved to the ./xfr folder (i.e., it is consumed).
- An attempt is made to perform site-specific scrub routines on the ofx file prior to import. This is done by matching the FID or BANKID value found in the ofx file vs those found in sites.dat site records.
- General (not site-specific) scrub routines are always applied to download files, in an attempt to correct known incompatibilities with Money (vs Quicken).
- A new SITE parameter was added named "delay". When defined, a delay will be inserted prior to sending a request to any account using that site.
- The value of the delay parameter is in seconds, and defaults to 0.0 (i.e., no delay).
Example:
<site>
AcctType : CCSTMT #credit card
fiorg : Discover Financial Services
fid : 7101
url : https://ofx.discovercard.com:443
ofxVer : 103
delay : 2.0 #delay 2 seconds
</site>
The new version can be downloaded here.
- Robert
New delay function works great! Thank you.
ReplyDeleteWith regard to creating accounts in Setup.py, can we accommodate a USAA password comprised of a static PIN number and a dynamic 6-digit security code generated by the Symantec VIP mobile App? How about 2FA for Wells Fargo with an RSA SecurID fob? Trying to implement your solution for the first time. Thanks.
ReplyDeleteThe OFX protocol doesn't support "3rd party" multi-factor authentication (MFA) methods. It does, however, support the use of a (mostly) hidden MFA method using a CLIENTUID. The clientuid MFA method is supported by the scripts if the ofxVer for a site is set to 103 or higher.
DeleteFor the scripts: A clientuid key is sent to the server to establish a trusted connection during setup. The key is unique for to that installation & account, and is stored in an encrypted file in the script folder (connect.key). Some banks allow the user to login to their site and run a special utility to "add a trusted connection". For Citi, the user starts the utility and then adds their account(s) in Setup (again, with ofxVer=103 for the site).
Trying to reach Robert Coleman, couldn't find another way...
DeleteI've just purchased a new Dell XPS 15 and was configuring all my software, including MS Money.
Everything came over beautifully, except none of the PocketSense seems to work. When I attempt to run getdata.py I get an error on ALL accounts saying ** An ERROR occurred sending POST request to ofx.fidelity.com (or whichever account)
Exception Type:
Exception val : [Errno 1] _ssl.c:503: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
I've tried to recreate them, test them etc, nothing seems to work... Any suggestions?
Did you install Python 3, rather than 2.7? I don't think the scripts work with 3.
DeleteAlong the same line... if using version 2.7.x, as required, make sure it's the latest version available. Older ssl versions won't be supported by most servers. I'm running 2.7.16.
DeleteI wonder if this is the thing where, on new (or newly wiped) computers, you need to log into each site via the web browser first (in order to get the sites' SSL certificate downloaded).
DeleteStill no success. I've logged into all the accounts on the new computer. I've completely reinstalled the latest PocketSense, and confirmed that I have Python 2.7.2 installed.
DeleteIn trying to re-add my accounts through setup.py, it never creates the account. I put debug statements in, trying to see what was going on, and the config_account module calls the client_doQuery which returns:
** An ERROR occurred sending POST request to eftx.bankofamerica.com
Exception type :
Exception val : [Errno 1] _ssl.c:503: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
That's where I'm stuck... So it seems to have something to do with SSL, but I have no idea what to do about that!
Python 2.7.2 is an *old* version. You need something much more recent. The ssl libraries changed around 2.7.13, iirc. I currently use 2.7.16.
DeleteAnonymous: (who posted August 5, 2020) said
Delete"Still no success. I've logged into all the accounts on the new computer. I've completely reinstalled the latest PocketSense, and confirmed that I have Python 2.7.2 installed."
Python 2.7.2 is not new enough. I use ActiveState Python version 2.7.12, and others use newer than that. So if 2.7.2 was not a typo, I have hopes that you get ActiveState 2.7.14 (I just checked that is available when logged in), that your SSL stuff will work. I am running 32-bit Windows, so that may be why I did not see a higher version.
Andrew Dingwall: I am quite impressed with your cookie postings. I don't need that at this time, but it is both interesting, and I am happy you have gained and shared that info.
Cal Learner
Robert,
ReplyDeleteThanks for the DELAY parameter. It seems to work like a charm!
Harold
hi, I replaced computer - trying to make everything works on new computer - reinstalled everything (new version) and getting this message when I run Getdata.py one every account:
ReplyDelete** An ERROR occurred sending POST request to online.americanexpress.com
Exception type :
Exception val : [Errno 1] _ssl.c:503: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
Can you please point me to rights direction to fix it?
upgrading python fixed it
ReplyDeleteI can not connect to Star One CU anymore - getting this in response:
ReplyDelete<_META NAME="robots" CONTENT="noindex,nofollow">
<_script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
<_body>
Anybody has the same problem?
Same issue here. Was able to connect previously and now nothing for past four months.
DeleteIs there a way to import Target qfx to MS Money? I've searched but unable to find a solution.
ReplyDeleteThanks in advance.
The simplest method is to change the name with an .ofx extension, rather than .qfx
ReplyDeleteThen by double-clicking the file, it should import itself.
Hi,
ReplyDeleteThank you for the quick reply. Unfortunately, I already tried that and didn't work. I also tried both removing the spaces in the OFX header and copying the qfx file in the import folder so that Pocketsense can scrub the file, but still didn't work.
See #1 in post above.
DeleteDo you mean #1 Import manually downloaded files from the list of updates? That's exactly what I did but still didn't work. There's no bank data for Target at ofxhome.com so I think Pocketsense is only doing a generic scrub and the resulting ofx still results in a corrupt file per MS Money.
Deletelate to the game but ... here goes. Target QFXs have a number of issues (bad tags, unmatched tags, header spacing) that make them not compatible with Money. I've written a scrubber but haven't had time to check it against the latest PocketSense. Once I do I'll pass it on.
DeleteTry this:
DeleteSiteName : Target Red Card
AcctType : CCSTMT #credit card
fiorg : Target
fid : 3820
url : https://rcam.target.com/ # not used - no ofx server
bankid :
brokerid :
appid : QWIN
appver : 2600
ofxVer : 102
mininterval: 31
timeOffset :
This entry just pulls the fields from the downloaded ofx (qfx) Target transaction search.
As Carlos says, their ofx is really bad, and will not import without major changes.
This post is in the category of a tip. I changed my broker password, and forget to change PocketSense. I did not have skipFailedLogon Yes. So I found that I had to reset my broker password.
ReplyDeleteI did that, and then went about belatedly changing PocketSense. At first I went to setup.py, but I got confused. "1. Add or Modify Account" presented a list of sites rather than a list of accounts. Choosing the broker, asked for a user name and password. At that point. I could have continued, but it looked like it would be a bit tedious-- even though I have copy and paste in command windows working.
I backed up and then edited ofx_config.cfg. Using a text editor, I changed all occurances of OldPassword to NewLongerPassword. I had edited ofx_config.cfg before with equal-length passwords, but the new one was longer. Anyway it worked on a test account. It was easy.
I then set up skipFailedLogon. I did testing and excuted my batch file that had failed when the scheduler had run it earlier. It pulled all of the accounts successfully.
CL
Cal,
DeleteYou must not be using the password encryption option (choice 5 in Setup), because I can't see my passwords in that file. Even though it worked for you, I think it's safest to delete the accounts, and then add them back in with the new passwords.
"You must not be using the password encryption option"
DeleteYou are correct. As an experiment, I copied my PS folder to a folder named PS_XX. In PS_XX I did #5 and added a password. Closed setup.py.
I examined ofx_config.cfg. Indeed, the password fields were encrypted.
Again, I did setup.py, and after entering the password, chose #5.
5. Remove Password Encryption
Selection: [0] 5
Remove file encryption and password protection (Y/N) y
Closed setup.py. Examined ofx_config.cfg, and found passwords were back to unencryped.
Then a strange thing. I again went back into setup.py.
I saw this:
----------Main Menu-----------
1. Add or Modify Account
2. List Accounts
3. Delete Account (or reset connection)
4. Encrypt account settings
6. Enable Stock/Fund Quotes
7. Test Account
8. About
0. Save & Exit
------------------------------
That was unexpected. I entered 5. Just got the main menu back. I exited setup.
I then did a FC of the ofx_config.cfg in the PS_XX with my regular folder. The files were identical. I have not tried rebooting. Maybe I will revisit that mystery at some later time. I don't have a need to use the PS password encryption.
CL
Duh... I was not thinking right. No mystery.
ReplyDeleteCL
I'm using the latest version of your script and I seem to be unable to get the script to work with Google Finance. This is what I used to try to pull the Dow Jones Index:
ReplyDeleteINDEXDJX:.DJI s:.DJI #Dow Jones (Google Finance)
I get the following error:
Searching .\import\ for statements to import
Getting security and fund quotes...
** error reading http://finance.google.com/finance?q=INDEXDJX:.DJI
** INDEXDJX:.DJI : invalid quote response. Skipping...
Any ideas on what I'm doing wrong?
Google Finance hasn't been working since they more-or-less killed the formal Google Finance site ~ mid 2018. Fortunately, Yahoo Finance was significantly improved about the same time, so the scripts were updated to better support the new Yahoo, which now includes support for a wide range of exchanges, including foreign.
DeleteI'll update the instructions and sites.dat template to reflect that Google Finance is no longer supported. I didn't remove it at the time, in case it was temporary, but then forgot about it.
Thank you for the quick response regarding the issue with Google Finance. The reason I was trying to use the site in the first place is that it seems to be taking 2-5 minutes per ticker symbol to pull the current price. When the script accesses my bank info, it only takes less than a second or so per account. Is there any suggestion you can think of that would speed up the quote gathering process? I'm running a relatively fast PC with a 1GBps internet connection.
ReplyDeleteThanks again!
That's odd. I have about five ticker quotes, and it takes about 1 sec for them *all* to complete. What version of Python?
DeleteJust started experiencing the slow down again yesterday. I obtain 85 quotes daily with probably a 20-30 second delay between symbols -- a lengthy process overall. Using Python 2.7.17 which I believe is the latest (and last) for the Python 2.x version. Any thought appreciated, as always.
DeleteAre you using the latest version of the PS scripts?
DeleteYes, dated 01-04-2020. Tonight update process took 21 minutes to complete the 85 +/- quotes on my list.
DeleteI made a slight modification to completely disable Google Finance, which is no longer supported. It may not be related, but worth testing. Beta version available for testing.
DeleteTried the Beta version with same results. It appears I am having significant issues with reaching the Yahoo quotes site -- loads of network timeouts which likely explains the slow quote responses. Are you aware of an alternate / alias URL that can be used to access their quote server?
DeleteYahoo has multiple site access points. For example, the Canadian version is ca.finance.yahoo.com . I haven't tested the scripts against those, but I'd guess that they use the same engine.
DeleteRunning the latest version. When I download the Quicken QFX from American Express (personal card) and place the file in the import directory it is processed and copied to xfr but the data is invalid in Money. When I look at the file in xfr it still has an xml header on it. Is anyone seeing something similar? I reran setup.py just to be sure I hadn't missed something. Same results.
ReplyDeleteI looked at another download (Chase) and tried to mimic the OFX header format from that, but that failed too.
Any suggestions?
Can you cleanup a copy of the xml message and send to pocketsense1/at/gmail/dot/com ? Remove or hash out anything personal.
DeleteThe Amex downloads have a problem where instead of <OFX> they have <OFX junk> where junk represents stuff to get rid of.
ReplyDeleteCL
You can script the connection to amex and it works great. Here is the settings
ReplyDeleteSiteName : AmEx
AcctType : CCSTMT #credit card
fiorg : AMEX
fid : 3101
url : https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload
bankid :
brokerid :
appid :
appver :
mininterval:
timeOffset :
I would avoid manually downloading anything from anywhere (in fact I have switched banks a few times over it). I currently download 12 bank/creditcard/brokerage statements daily with never having to log into anything but ms money (and I do have a password on my script also). Bravo PocketSense!
Thanks for sharing. I updated the url on my Settings page: https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
DeleteThanks. I had the same url and fid, but had also filled in values for appid (QWIN), appver and mininterval, probably from some point in the past where AmEx looked at these. Making my site match yours restored the auto-download.
DeleteI will still put together a copy of the download file and send it to Robert, in case AmEx breaks the auto-download in the future. Appreciate the pointers everyone!
Just found this and am blown away! One quick question that I can't find addressed elsewhere: I get an error when testing a USAA account "OFX message contains ERROR condition". Is this because USAA requires a username, password AND PIN? If so, how do I fix it?
ReplyDeleteThanks so much for this!
I don't use USAA, but if you look at the server response in the ./xfr folder, it may shed light. The OFX protocol uses a different type of multi-factor authentication. If you haven't, try setting ofxVer=103 for the sites.dat entry, which enables using a unique client key.
DeleteHello, Robert -
DeleteI have just discovered your add-on for MS Money; this is very cool and let me also say I'm as grateful as Jimbthepilot. I, too got the "OFX message contains ERROR condition" during setup of a USAA account but then after adding 'ofxVer=103' to the sites.dat entry as you suggested, I get a different error right after entering username and password:
"An error occurred requesting accounts from this site. Please check username and password."
USAA requires 2-factor authentification; could this be the issue?
Many thanks in advance!
The only MFA method supported by OFX (direct connect) is the client connect key, which ofxVer=103 enables. Some banks require the user to have direct connect enabled for OFX (Quicken) connections to work. Is this a bank account? If so, remember to enter the bankID as the routine number.
DeleteDid you see the last comment posted on the ofxhome entry from last year, or other comments for that matter. May help?
Hopeful USA user: I don't use USAA. You have not got a response, so far, from somebody who does.
Delete1. General tip: When you are referring to a financial instituton that offers credit cards, banking, and broker accounts, it would be good to know which you are using. Somebody who just uses the credit card may not know what it takes to set up for the brokerage account.
2. When you have tried to set things up in PocketSense, it would be good to show the contents of your site descriptor. That is in sites.dat for each financial institution. The site descriptor scriptor would be the lines between the <site> line and the </site> line for a particular FI.
3. When you are trying to diagnose an error, look at the *.ofx file in the xfr folder. Use a text editor such as Notpad.
Find the <SEVERITY>ERROR. Then usually following that is <MESSAGE> Then following that are some words that try to explain the problem. What are those words?
4. While I don't use USAA, I do find some discussions that seem like they could help. Robert pointed to one. Another is https://wiki.gnucash.org/wiki/OFX_Direct_Connect_Bank_Settings#USAA What that seems to say is to not pad the 7-digit USAA account number with zeros. That may be old info.
5. When you do try your connection, look for any website messages or emails from USAA. They may ask you to vouch for the request they just got.
Cal Learner.
I have accounts at USAA for banking, credit cards, and investments. The PocketSense scripts will work if you do these two things:
Delete1) Set your PocketSense UserName to your 7 digit USAA number rather than your alphanumeric USAA OnlineID.
2) Set your PocketSense password to your 4 digit PIN rather than your USAA login password.
Hope this helps.
David
Hello again Robert and to all on this thread; SUCCESS!!! Thank you, thank you! :-)
ReplyDeleteI posted as Anonymous on February 9, 2020 at 4:36 PM regarding problems I was having setting up a connection with USAA. After following the feedback and suggestions, and with a little patient trial and error tinkering with the Sites.dat and Setup.py files, I stumbled on the solution that in retrospect was deceivingly simple.
First of all, for the SITES.DAT file use the following parameters:
#--------------------------------------------------------------------------------
SiteName : USAA
AcctType : BASTMT #bank statement
fiorg : USAA
fid : 24591
url : https://service2.usaa.com/ofx/OFXServlet
bankid : 314074269
brokerid : 5874
ofxVer : 102
appid : QWIN
appver : 2400
mininterval:
timeOffset :
Then, under SETUP.PY, #1. Add or Modify Accounts
USERNAME = your USAA member number (do *NOT* pad with zeros)
ACCOUNT PASSWORD = your USAA account PIN
Following these inputs the USAA website is pinged and then a menu of your account numbers will download; select the account you want to continue setting up. Don't forget to SAVE at the end of the setup routine before closing out.
Thanks so much for all of your help so I can go back to using MS Money again. Happy Day!
David Casada on March 5, 2020 said "I have accounts at USAA for banking, credit cards, and investments."
ReplyDeleteDavid, what do you use for the USA sites.dat descriptors for credit cards and investment?
Cal Learner
I've had my Citibank VISA statement imports popping up the account selection dialog recently, after years of it importing into the proper account automatically. Anyone else seeing this? I run a money-popup window catcher tool I wrote to auto-click most of the common dialogs that come up, but this is one I intentionally leave alone. Busts my daily automation.
ReplyDeleteI wrote about the solution in my blog: https://microsoftmoneyoffline.wordpress.com/2020/02/29/fixing-citicard-downloads/
DeleteHello,
ReplyDeleteI have a recent problem where my stocks are receiving an older price from one of my accounts. The Pocketsense stock quotes works perfectly. When I run my script (with stock quotes either first or last) my stock prices get updated with the account data and not Pocketsense's quote.
Is there anyway to make Pocketsense's quote primary? My current workaround is to run Setup and test stock quotes -> import to Money, then run my regular account getting script.
Andrew:
DeleteIt is possible that the quotes your broker downloads are running into the problem tht the timeOffset is intended to fix. I have not played with that for a while.
If worst came to worst, we could come up with a scrubber that removes all quotes from the OFX but not the transactions.
You might identify the broker and the timeOffset you are using.
When looking back at history of a stock, I think no quotes should be dated Saturday or Sunday.
Cal Learner.
Another thing to consider is the "combineOFX" option. I'm not sure how Money processes multiple statements in one file, i.e. whether it forces order, as I never tested it. If you run the script with combineOFX=No and SaveTickersFirst=Yes, does it still do it?
DeleteGents:
DeleteI've discovered some odd behavior. When I downloaded quotes this morning I experienced the issue. I saved that combined.ofx file and pulled this info from the combined SECLIST at the bottom of the file. This is the entry from Pocketsense. The other entries from USAA are there but give the correct price of 231.16.
<STOCKINFO><SECINFO><SECID><UNIQUEID>IVV<UNIQUEIDTYPE>TICKER</SECID><SECNAME>iShares Core SP 500 ETF<TICKER>IVV<UNITPRICE>230.99<DTASOF>20200321102400[-5:EST]</SECINFO></STOCKINFO>
I tried to reproduce the behavior today and this is the equivalent entry in the combined.ofx file but it now has the correct price (the other USAA entries are still there and still correct).
<STOCKINFO><SECINFO><SECID><UNIQUEID>IVV<UNIQUEIDTYPE>TICKER</SECID><SECNAME>iShares Core SP 500 ETF<TICKER>IVV<UNITPRICE>231.16<DTASOF>20200321081500[-5:EST]</SECINFO></STOCKINFO>
Could I just be getting stale data from Yahoo? At 1000 the day after the stock market closes?
-Andrew
Do the scripts work for Charles Schwab? I opened a brokerage and checking account earlier in the year and I have seen conflicting things on my google searches about it working or not. Sorry if this info is covered elsewhere I did some blog searches without finding anything.
ReplyDeleteThe scripts work great for Schwab. These are my settings that also work for the Schwab1 account:
DeleteSiteName : SCHWAB
AcctType : INVSTMT
fiorg : ISC
url : https://ofx.schwab.com/cgi_dev/ofx_server
fid :
bankid :
brokerid :
appid :
appver : 2800
mininterval :
timeOffset :
ofxVer :
Make sure you go into Security Center Settings while logged into your web access, and Enable Third-Party Access.
ReplyDeleteRobert,
ReplyDeleteI've discovered an issue with the Null transaction scrubber, which may be caused by an unexpected tag within INVBANKTRAN. I've written a description and provided screenshots of the issue in my article today at https://microsoftmoneyoffline.wordpress.com/2020/03/27/null-transactions-scrubber-issue-schwab-pocketsense/ but if it isn't easily fixable, don't worry about it - at least it will now be known.
Dan: Just now looking at this. I primarily added the skipZeroTransactions option for credit card statements, since some add $0.00 transactions just to record extra comment lines. Perhaps it makes sense to change this option such that it applies only to CC statements?
DeleteYes, that would be good, if it isn't difficult for you. Had it happen again yesterday, so I had to turn the option off again. TIA
Delete-Dan (ameridan)
Hi Robert. I am just trying to use your brilliant script to import the quotes into my newly creates "Quotes" account. I don't need to connect to any institutional accounts. i got the quote bit working fine, but completely foxed about setting up the "Quotes" account to receive the data. If I run Setup i see option 1 is add or modify accounts. I enter that and see a list 1. DISCOVER 2. USAA 3. VANGUARD 0. EXIT. Now what? I tried entering 4 which gave me main menu again. How do I add my dummy account "Quotes"? Thanks for any help mike
ReplyDeleteYou manually create the dummy account in Money, not Pocketsense.
ReplyDeleteyes i did that, but dont i need to tell pocketsense where to download the quote data to? When I ran the getdata tp run test data i got this
ReplyDeleteDownload transactions? (Y/N/I=Interactive) [Y] y
Download interval= 7 days
No accounts have been configured. Run SETUP.PY to add accounts
Searching .\import\ for statements to import
Press to continue...
In setup.py, which do you see?
ReplyDelete6. Disable Stock/Fund Quotes
6. Enable Stock/Fund Quotes
Cal Learner
ah got it work i think now thanks
ReplyDeleteStil not quite right
ReplyDeleteI run setup and click 6 to enable quotes. Y to test quotes all looks fine. y to send to money and money imports data and asks me if the quote relates to existing instrument or create a new one. That all looks fine listing out the 69 instruments and quotes. however when i go to portfolio none of my instrument prices have actually updated in the existing accounts? Is there another step? Thanks
Also might have messed up a couple of quotes associating them with a name that is slightly different to the one in my portfolio. Where can you go to edit the association of say "AFMC" ?
Unused stocks/fund entries can be deleted from: Portfolio --> Work with investments --> Delete an investment from all accounts
Deletetheferret: Try this: In Money Portfoloo, right-click the investment that should not have the quote, and blank or fix the symbol. Right click the investment that should have that symbol, and put that symbol into place.
ReplyDeleteCal Learner
I must being doing something wrong with this. I have a holding of BSRT in an account "Mike". I have also added a dummy holding to account "quotes" bought at a price of 4. so my real holding is now revalued at 4.
ReplyDeleteam I meant to run Getdata or setup to just get quotes? Getdata asks me download transaction and if I say yes I get "no accounts have been configured"?
If i run setup I have press 6 to enable quotes. then run test download and BRST price is 0.451. Send results to money, import complete . Review investment holding shows BRST as 0.451. Click next to update Money.
now I get a question "thew positions reported on 3/4 statement do not match those in your money account (as I added it manually). Choice to remove the shares in the account or not. I pressed no. go to "quotes" account and price is still 4. So price update is not working at all.
I can't see what I can do differently as looks like its doing the right thing but just isn't!!!
Appreciate any help
Hi,
ReplyDeleteI have just started to use the script for importing the quotes and it works well for all the quotes that I can find within Yahoo Finance. However since I am based in Asia, most of the Asia mutual funds are not available from Yahoo or Google Finance. Is there a quick way to modify or add the script to use https://www.bloomberg.com/quote to retrieve quotes on funds that available at Bloomberg.com?
I posted back in March 23 asking how to set up script for Charles Schwab. From the reply I received I set up the script. I have an investor checking account and an associated brokerage account. When I try to add the account through setup I am getting a check user name and password error. I know I am entering the correct username/password combo because I can log in through the Schwab web site. Can someone advise what the problem might be?
ReplyDeleteI have my script set up as follows:
SiteName : SCHWAB
AcctType : INVSTMT
fiorg : ISC
url : https://ofx.schwab.com/cgi_dev/ofx_server
fid :
bankid :
brokerid :
appid :
appver : 2800
mininterval :
timeOffset :
ofxVer :
This comment has been removed by the author.
DeleteSee March 23, 2020 at 3:03 PM comment above.
Delete@NxtTek Thank you I'll check on that
Delete@NxtTek That was my issue thank you.
Delete(If it's Monday, it must be Discover-not-working day)
ReplyDeleteThis morning, all my Discover accounts came back with "You don't have permission to access "http://ofx.discovercard.com/" on this server.-P-
Reference #18.acfa3b17.1586189288.17b14e14"
This affected both the Discover credit card and bank accounts. Can anyone interpret the message above? (I changed the characters framing the letter "P" to allow the message to post. they were originally angle brackets.)
Hi Harold,
DeletePython 2.7.14
Python Scripts, Ver: 07-Mar-2020 BETA
D.C. is broken in Moneydance as well as the scripts.
MD is reporting a '403 Forbidden' response along with the "You don't have permission..." response.
-Kevin N.
Thanks for letting me know. In this case, though, misery does not necessarily love company!
DeleteThey just don't quit... ;)
DeleteI added the import feature last time Discover stopped working, so I'd suggest using that until we have more info.
Anyone have more info? Been two weeks now...
DeleteI've been using Money since it came out and love both it and PocketSense. My Money file was over 80MB in size so I did some cleaning up and got it down to around 52MB. I have three Fidelity accounts in there with data going back to 1999. I would love to purge all transactions in them before 01/01/2019. Archiving doesn't do that. Could I unlink the accounts, set up three new accounts and do an import from 01/01/2019 and then delete the old accounts? Any suggestions would be appreciated. Thanks.
ReplyDeleteRather than delete ALL transactions before 2019, it might be best to leave the cash transactions alone (so transfers to/from other accounts and paychecks aren't all messed up), and in the Investment Transactions register, just start deleting. Then when done, re-balance the Fidelity account values.
ReplyDeleteJoe@TheWho.org:
ReplyDelete1. I doubt that deleting the old buys, sells and dividends will have much effect on file size. Deleting those transactions will not get rid of the old quotes, which take space
2. A 4 gb flash drives, a small one today, will hold a lot of 80 meg drives plus a lot of compressed backups. My Money file is 240 MB.
Cal Learner
I'm not so concerned about the file size. My data drive is 3TB. However, the Money program itself just seems to be getting slower and slower. Thanks.
ReplyDeleteJoe:
ReplyDeleteAh, yes. If you have not done so, use the Windows Firewall to block Money from waiting for servers that are no longer there.
Also, if somebody's file is *really* slow, it is possible that the file has slow file syndrome, which is cured by removing all bills. Maybe try that on a copy of your file.
To get some information within the program, File->Repair->RemoveAllBillsData
Read the information. Then click cancel.
Before proceeding, make an extra copy or backup of your file in case you want to undo something.
The problem will be that if you remove all bills, you will probably
want to add at least some of your bills back. This can be done
manually with the aid of the reports that you print out before
removing the bills.
A technique that could help simplify re-entering your bills
afterward would be to go your register and selecting a
representative instance of your past bill and selecting "Make
Recurring" (Ctrl+E). This would be especially helpful if the bill
is complex, such as involving a split.
I am not sure what is the limiting hardware speed-- processor overall, single thread performance, drive speed? I suspect drive speed is a biggie. Not graphics speed.
Cal Learner.
Thanks for the suggestions, Cal. When I load Money and it updates Bills it only takes around 5 seconds. However, Internet Update takes several minutes. When I look at the results it says "Your last call encountered an error: 04/03/2015 11:04 AM" and "An error occurred while Money was retrieving quotes, news, and exchange rates. Please try again later." And then when I look at Downloaded Statements there are some accounts with old Last Downloaded dates. For example, Fidelity Visa 12/04/18, Hilton Amex Business 11/13/2018, Line of Credit 05/20/2106. How can I stop those updates?
ReplyDeleteI removed all of my accounts from Online Services. File size is now 27MB and it loads up very quickly.
ReplyDeleteJoe: Intriguing. Are you using Sunset? I think at least some of that Internet stuff was removed/disabled from Sunset.
ReplyDeleteWhen you removed the accounts from online services, was that on the "Manage online services" page?
Cal Learner
Cal: Yes, I am using Sunset. I remember when I first started using Money it was free and came on a floppy.
ReplyDeleteI did remove them on the "Manage online services" page. At least now that little "Online updates" wheel in the top right corner doesn't spin on startup.
Still, the program takes 45 seconds to open. Interestingly, I tried a file repair and the repaired file was 60MB. The original file is 27MB.
I'll keep tweaking.
Joe:
ReplyDelete1. If you have not done so, use the windows firewall to block Money from trying to access servers that are no longer there, and then waiting for them: https://microsoftmoneyoffline.wordpress.com/2016/06/06/eliminating-the-online-updatingdelays-and-errors-when-opening-money/
2. Make sure you have installed the Money Plus patch. This is not a speed thing, but a misoperation thing. https://social.microsoft.com/Forums/en-US/d28d6921-1f01-430f-b33e-f5a3ec47077b/important-patch-for-money-plus-maybe-20062007-mnyob99dll?forum=money
Cal Learner
Thanks, Cal. I did both things and it still takes 45 seconds to open. I created another file named My Money Test.mny and it opens instantly. I'll try the Bills cleanup thing.
ReplyDeleteCal: I did the Bills cleanup thing and it now starts instantly. Now to rebuild them. Thanks for all of your help!
ReplyDeleteI just discovered PocketSense after getting frustrated with Quicken Home & Business 2017. It is just slow after awhile of using it for some reason. I got Microsoft Money Sunset, opened up my old Money file from 2006, imported everything from Quicken into it, and now everything is nice and fast.
ReplyDeleteThe only problem I am having (I know it is an old problem that comes and goes) is I cannot download my Discover CC transactions. I also cannot download Capital One transactions. Am I doing something wrong? I have my American Express and Chase CC accounts and Charles Schwab brokerage account sat up and everything is working great so far with PocketSense:
SiteName : Capital One
AcctType : CCSTMT #credit card
fiorg : Hibernia
fid : 1001
url : https://onlinebanking.capitalone.com/ofx/process.ofx
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval:
timeOffset :
SiteName : Discover
AcctType : CCSTMT #credit card
fiorg : Discover Financial Services
fid : 7101
url : https://ofx.discovercard.com:443
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval:
timeOffset :
delay : 2.0 #delay 2 seconds
I don't use Capital One, but maybe someone will chime in. As for Discover, the connection recently changed (er, broke... again...), but you can download statements from their site (qfx format) and save it in the .\import folder. See the post entry for this thread (above) for how import works.
DeleteManual downloads from Discover Card need a different Sites.dat entry, to ensure proper matching for the scrub routine. Add the following to your sites.dat file:
SiteName : DISCOVERCARD_Import
AcctType : CCSTMT #credit card
fiorg : Discover Card Account Center
fid : 9625
url : https://ofx.discovercard.com
The Discover *bank* entry in sites.dat is the same for both the ofx and manually-downloaded statements.
The Discover connection has been changed or broken numerous times, so I now wait a while before trying to resolve it, to see what info comes available.
I'm in the same boat. About 10 days ago my automatic downloads for Discovercard stopped working. I deleted the entry in sites.dat and then went to add it back. When I put in my ID and password I got the message "An error occurred requesting accounts from the site. Please check username and password." Has anyone else had this problem recently and then gotten automatic downloads to work?
DeleteThis comment has been removed by the author.
DeleteI've never actually been able to get Capital One and I've had it for years. Let us know here if you do get it to work.
DeleteHi Hoping someone can help me as really getting no where with this. Having failed to get the quotes actually turning up in my old version of Money 2005 i though I would try installing the sunset version and see if that worked any better. Being terrified of wrecking my existing installation i installed the sunset version on another machine. I put my current money file on the c drive and tried to open it, but it just says cant open file as created in a different version of money? Earlier i had it saying it was going to do a conversion, but then couldn't complete that. I tried to repair the existing file, but just made another old style file the new program cant open. What am i doing wrong that i can't get it to convert my old file to the new type? Thanks for any advice. Mike
ReplyDeleteI didn't catch that you were using a non-US version before. If you look in sites.dat, you'll see an option near the bottom named "ForceQuotes". If you enable that option, it should force Money to record quotes. It's been a long time since I added it, and don't recall the details, but it seemed to work. Read the notes about it just below, or in the 10-Sep-2013 entry in the revision history.
DeleteHi Robert. Sorry i'm making a complete pigs ear of this. i found that setting and changed it to yes. I tried to rerun setup and import quotes to money but when i send results to money I now get "this file does not have an app associated with it...." I'm guessing i broke something when messing about with the sunset edition edition. I tried reunzipping the files to get a new version but same error. Can i fix this or done some permanent damage? Thanks
DeleteSee FAQ #5
DeleteThis comment has been removed by the author.
DeleteI ended up reinstalling money as somewhere had corrupted the ability to import the ofx. Think I installed sunset version then uninstalled which i now read knackers it. So i am back where i started. i have set the forcequotes option in site.dat to yes, but still same thing. I run setup get the quotes, send to money, money asked which account, put them in "Ticker" account, says import complete and a list of all the tickers with new price. However none of the prices in any of my accounts has actually updated?
DeleteTicker symbols are exactly the same?
DeleteYup. i have entered the data in sites.dat like this
DeleteAAIF.L m:0.01 s:AAIF
ABD.L m:0.01 s:ABD
AEMC.L m:0.01 s:AEMC
AFMC.L m:0.01 s:AFMC
AGT.L m:0.01 s:AGT
AJIT.L m:0.01 s:AJIT
ALAI.L m:0.01 s:ALAIF
When I import it takes me to the the downloaed statement page so I see Tickers (73) 14/4/20
If I double click this i see a list where the symbols match mine, but the security name is not the same? For example ABD in my money is "Aberdeen New Dawn" but in the import is "Aberdeen New Dawn Investment Trust PLC". I don't know where this other name is coming from guess that is Yahoo name for it? If I try to add a new buy I only see my name listed. I'm assuming the symbol is the unique identifier? If i go 2add an investment" and type in symbol ABD it comes up with "Aberdeen New Dawn" so pretty sure my symbols are consistent.
I'm not sure then. I checked, and the names don't always match in mine, but if the symbol matches, then it updates the price. I can edit/change the name of any security in Money. If you look at the .\xfr\quotes.htm file, are the prices/symbols correct there?
DeleteAlso, are you getting this message when running GetData?
"ForceQuote statement loaded. Accept in Money and press to continue."
Hi Robert. Yes looks correct in the htm file. Basically it gets as far as doing the import to money and i see al the right prices in the downloaded statement table, but when I hit finish it does nothing.
DeleteI dont know if the problem is i am using Setup.py to get the data? If i run Getdata it says "download transactions". if I put I it says no accounts have been configured. i didnt configure any accounts as i'm not downloading any transactions. Even if i wanted to going into Setup.py and choosing 1 Add or modify i see a site list 1. discover 2.usaa 3. vanguard, but dont know what i am meant to do to remove these and add something (no idea what i'm adding as don't have any accounts)? If I enter 1 it asks me ofr a username and password for discover which I dont have. i tried using 7. test Account then 1. fund/stock prices then test pricing update, but this does exactly the same as going through setup. Do I need at least one account setup? Thanks
Enable Quotes in Setup.py (Option 6) and then use GetData to get quotes. The "forceQuotes" option only works from GetData.py.
Deletep.s. No need to add an account.
DeleteOk so can you just tell me what the steps are to use getdata if i have no accounts setup. The only way i've got it to work gives the same result? Thanks
Delete1. Enable forceQuotes in sites.dat
Delete2. Enable Quotes in Setup.py (menu option 6). When enabled, it will change to "6. Disable Stock/Fund Quotes". I guess that could be confusing...?
3. Open Money (must be open if using forceQuotes)
4. Run GetData.py
You should be prompted to accept the quotes statement TWICE when forceQuotes is enabled.
So when I run Getdata it asks download transactions? If I put Y or I it just says "no accounts etc etc". So I cant get any further. How do I get round this please?
DeleteReally not sure. I just tested a new install, with no accounts, and it worked fine. Make sure that your stock symbols are defined between the <stocks> ... ... ... </stocks> tags. When you run Setup, does it show "6. Disable Stock/Fund Quotes" ? Notice it must say "disable", *not* "enable".
Deleteah cracked it. When you hit 6 to change to enable you have to also hit 0 to save the setting!!!! Thnik it is now working. Thanks for your patience and help
Deletetheferret: Sunset can only open or convert North American Money files.
ReplyDeleteIt is possible to convert via QIF export from Money 2005 UK, and import into Sunset, but there are some limititions and chores to make that happen well.
If Money Plus created a *.m14 file, that is a copy of your Money 2005 file before conversion was attempted, and the *.mny file would not be able to be opened in Money 2005.
Cal Learner.
Ah Didn't appreciate we Brits were left out in the cold!!! Thanks for clarifying.
ReplyDeleteSo assuming i'm stuck with my old Money 2005 is there some reason when I import the quotes using pocket sense that they just don't actually affect any of the price of my instruments? Everyone else seems to find this trivial!!!
See reply to your original post above.
DeleteI have a little quirk happening since I've cleaned up my Money file. When I run the import my three Fidelity IRAs import OK. However, I have an investment account with Fidelity and every time Money says "Select a Money account for your imported file." I select the account from the drop down list but it still does this every time I import. Thanks!
ReplyDeleteJoe:
DeleteTry it using test account twice in a row. Still happens? No error indication?
Confirm that the <ACCTID> is the same each time for that account.
I don't know if it matters, but check that the Account Number in the Money Details for the account matches the account number.
Cal Learner
Cal: The first time I test the account and send the results to Money the window pops up asking if this is the right account. The second (and third) time I test it it works properly. But then when I run Getdata.py the window pops up again every time. I even tried setting up a new account.
ReplyDeleteThe account numbers do match.
Thanks.
I directed it to a new account named Fidelity and it works properly. However, when I deleted the old account named Main it then asks where I want to direct a Broker account XXXXXX6789. If I direct that one to the new Fidelity account the problem comes back.
ReplyDeleteI notice if I look at the combined.ofx file with Notepad the following:
PocketSense012345678920200413080600[-5:EST]
Joe: change any < symbols to < before posting on this site.
DeleteChange any > symbols to > before posting on this site.
CL
Joe: if you really are getting XXXXXX instead of part of the account number, I have a modification that you could make to the scrubber for Fidelity. That has occured with some other financial institutions, but this is the first time I have seen it reported for Fidelity.
DeleteThe following is for somebody who is familar with program modifications. It could be polished up. Maybe Robert will even incorporate into a future version. This is to substitute the real account ID for the munged-up version of the account ID.
ReplyDeleteI uploaded ps_subs.py to
http://s000.tinyupload.com/?file_id=53546724146835642331
File will disappear if not downloaded for a while.
ps_subs.py file can be run and does a little stand-alone coding test. You don't need to do that.
The d tuple needs to be modified for your purposes. In the example uploaded, there are 4 strings tested for, and a match will replace the first part with the second part.
The purpose is to replace the munged up account number(s) with your actual numbers. 4 lines are shown, You can add or delete lines to match
your needs.
The expectation is that you would put this line in scrubber.py:
from ps_subs import make_substitutions
This was first written for Citicard, but could be use for any account number substitutions. The next pararaph says Citicard, but replace that with the FI you are dealing with.
Then have this line run as Citicard gets scrubbed:
ofx=make_substitutions(ofx)
Cal Learner
Thanks, Cal. I just pointed the update to a different file and it works OK for now. I do appreciate all of your help very much. My Money program is really speedy now.
ReplyDeleteHi, Robert....
ReplyDeleteRe, Discover: Can you take a quick look at the April 25th entry at https://infinitekind.tenderapp.com/discussions/online-banking/15756-discover-appears-broken-again ? What he says there does not seem to work for both DiscoverCard and DiscoverBank, but it may need a programming to change the order that is discussed.
Here's a more direct link to the post that I believe Harold is referring to.
Deletehttps://infinitekind.tenderapp.com/discussions/online-banking/15756-discover-appears-broken-again#comment_48281568
PocketSense has already sent the headers that way since the March 11 , 2017 version. I spent some time this morning verifying that we still send http headers correctly to Discover. I also tried the alternate FID and ORG in a request, and resetting the connection. All receive the same 403 Forbidden reply.
DeleteYou can see what's sent by setting Debug=1 in the doQuery function in ofx.py. Here's what is sent for http headers,
HTTP/1.1
Content-Type: application/x-ofx
Host: ofx.discovercard.com:443
Content-Length: 716
Connection: Keep-Alive
followed by the actual ofx request containing the user, password, ...
Robert,
There is a small bug in the Reset Connection logic in Setup.py. When explicitly resetting a connection, the account isn't deleted and thus will be found. Also, urlHost is undefined in the print statement following.
Thanks for following up Andrew. As a fyi, the "reset connection" only clears (resets) the (2-factor) client connect key. The "delete account" option is the only way to delete, which also clears the key if no other connection uses it. Print bug fixed for next update. Also, the Debug flag can be set globally in control2.py.
DeleteThank you for this wonderful script. I've tried this out for stock/fund prices and I've found that if I import in the morning then try to import prices in the evening, the first set of pricing isn't replaced by the evening's set. Can you confirm that this is true?
ReplyDeleteThat is a MS Money behavior. Only one set of quotes is accepted per date. I run PocketSense once per day just after the market close to pick up the stock quotes and fetch statements.
DeleteAnyone using PS to download from Wells Fargo?
ReplyDeleteI'm having success getting Vanguard and Schwab statements, but WF is a no go.
Can anyone post the settings if this is possible?
Many Thanks,
an
Tim, yes that is the expected behavior. I've written a few articles on that subject:
ReplyDeletehttps://microsoftmoneyoffline.wordpress.com/2010/05/03/updating-quotes-via-ofx/
https://microsoftmoneyoffline.wordpress.com/2010/03/20/can-anyone-explain-why-quotes-are-handled-differently-between-quotes-py-and-ofx-py/
but primarily read the 1st paragraph of this article:
https://microsoftmoneyoffline.wordpress.com/2014/05/08/ms-money-quotes/
- ameridan
Long time user for many years, thank you Robert and to all who support this community.
ReplyDeleteI’ve (finally) succeeded in getting MS Money Sunset US to run in wine on a Mac, I don’t need VirtualBox anymore!
Just wanted to share some notes I made to the scripts.
1. I changed del to rm in Getdata.py:
#delete old data files
ofxfiles = xfrdir+'*.ofx'
if glob.glob(ofxfiles) <> []:
os.system("rm "+ofxfiles)
2. And in rlib1.py it would throw a permission problem when trying to upload the combined ofx to Money:
def runFile(filename):
#encapsulate call to os.system in quotes
#os.system('"'+filename+'"')
subprocess.call(('open', filename))
return
This can be automated for each OS type with something like this:
import subprocess, os, platform
if platform.system() == 'Darwin': # macOS
subprocess.call(('open', filepath))
elif platform.system() == 'Windows': # Windows
os.startfile(filepath)
else: # linux variants
subprocess.call(('xdg-open', filepath))
If anyone wants to try this, you have to be on Mojave or earlier. Catalina doesn’t run 32-bit code so that will be a showstopper; I did have to downgrade.
You have to do some homework/reading if you haven’t messed with Wine before, but it’s not too bad.
For reference: http://wineskin.urgesoftware.com/tiki-index.php?page=Manual
When you’re ready to give it a spin:
- Go to https://github.com/Gcenx/WineskinServer/releases/tag/V1.8.4 and get one of the Unofficial Wineskin Winery files.
- Update the wrapper version, currently at 2.9.0.6. Build a new blank wrapper using engine WS11WineCX19.0.1.
- Using winetricks, install into the wrapper corefonts, Tacoma, MSXML 4.0, IE7, vcrun2005 (Visual C++ 2005 Runtime Libraries).
- Install MS Money Sunset.
This isn’t for the faint of heart / tech-challenged so I’m not posting a detailed step-by-step. I am happy to answer questions though.
Nathan
Does anyone know if Marcus by Goldman Sachs has a way to interface to Pocketsense?
ReplyDeleteHarold. There is no way to interface Marcus by GS with Pocketsense. You cannot even download an OFX file.
ReplyDeleteCL
I found these settings for Goldman Sachs in Moneydance. Has anyone tried them?
ReplyDelete"uses_fi_tag" = "y"
"access_type" = "OFX"
"fi_name" = "Goldman Sachs"
"bootstrap_url" = "https://portfolio-ofx.gs.com:446/ofx/ofx.eftx"
"fi_id" = "1234"
"app_id" = "QWIN"
"fi_org" = "gs.com"
"broker_id" = "gs.com"
"app_ver" = "2400"
Chase seemed to not like any of my regular Chase bank accounts this morning (the Chase credit card accounts worked just fine). The returned OFX file had a code 2000 error with text that said "[ofx.validationException]". Do you think it's a transient error, or something that needs attention?
ReplyDeleteProblem seems to have cured itself.
DeleteSo have we given up on Discover?
DeleteSo have we given up on Discover?
ReplyDeleteIn case you missed it (April 13th comment @ https://pocketsense.blogspot.com/2019/11/new-version-available.html?showComment=1586784196272#c4153592209771088476 ), Robert has provided an alternate, which involves periodically grabbing the download from Discover's website. I got a Citi DoubleCash card now (2% cashback vs Discover's normal 1%) that gets most of my business now, as a form of boycotting Discover Card :)
ReplyDeleteThanks... I don't really understand the alternate. If I have to manually download the file from Discover's website, why won't I just open that file after downloading so it imports directly into YNAB (I use YNAB instead of Microsoft Money). For me, the value here is not having to manually download. After manually downloading, importing is faster by double clicking vs putting it into an import folder etc. Maybe Money is different though and that makes the import folder useful.
ReplyDeleteRo: The advantage of passing the OFX via PocketSense is to allow a scrubber to potentially modify the OFX file. There are some geneneric functions in the scrubbing, and there can be site-specific scrubbing actions.
DeleteCal Learner
Because you use YNAB, you are correct. Double-clicking works for us too, but by placing the file in the import folder, it is automatically processed and scrubbed, and file deleted afterwards. The scrubbing makes the transactions match the former downloads, in case that ever works again. Otherwise each transaction will appear different, and the doubles would have to be removed.
ReplyDeleteYeah, I've never gotten doubles from importing using PocketSense and then manually downloading. I've done that from time to time with my accounts. YNAB handles that correctly i.e. tries to prevent duplicates. In fact, when I have 2 actual transactions of the exact same size, account and date, YNAB only imports 1 of them (it assumes the 2nd is a duplicate). As such, putting in the import folder isn't necessarily useful to me.
DeleteYou're right about matching the former downloads because in YNAB, the first time you manually download, you have to select an account. When you use pocketsense, the account can be different. However, you only ever have to do select an account one time for each method so not a big deal either. Still ends up being much easier to download and click on it in my Chrome browser vs having to put it in a folder then run a program to get it into YNAB.
Regarding Discover, I'm of the current opinion that they will no longer support direct connect. Some online discussions (Moneydance, etc.) indicate that they will only support Quicken going forward, which is odd considering it's a dying platform. Sites like Personal Capital are able to connect, but they may be using "screen scraping" methods, with the corresponding (convoluted and high) maintenance.
ReplyDeletePersonally, I'm using the download option and letting the script import fix the (known) issues. It only takes a minute a week, so not a big deal. Even so, I'll be looking at other options when I have time. I don't begrudge their decision, as it's their prerogative, but I'm more interested in my convenience than theirs ;)
p.s. I wouldn't say that I've "given up"... just not especially hopeful.
DeleteNormally, the way to make this work is to see exactly what Quicken is sending to Discover and send them exactly the same thing. I guess that Discover could intentionally defeat this approach, but it would seem that they would almost have to "work hard" to do so. Why does this approach not seem logical at this time?
DeleteWindows 10 Ver. 2004
ReplyDeletePS Ver. 07-Mar-2020 BETA
Python 2.7.14
Fidelity Investments started throwing an error:
Access Denied You don't have permission to access https://ofx.fidelity.com/ftgw/OFX/clients/download on this server.
This started last night 06/11/2020.
Is anybody else seeing ths?
-Kevin N.
@Kevin N.
ReplyDeleteI am getting a Fidelity error that says Invalid OFX Statement Detected. I hope this gets fixed Fidelity is a major holding for me.
Kevin N: Sadly, I get the same. It was that way in my automated fetch this morning, and it was that way just now when I did a test in setup.py.
ReplyDeleteCL
Kevin N: I get this in the OFX in the xfr folder:
DeleteAccess Denied
You don't have permission to access ...
Cal Learner
I just tried the old Discover Card trick, adding :443 to the url. It works.
DeleteIn sites.dat, set the url for Fidelity Investments to
https://ofx.fidelity.com:443/ftgw/OFX/clients/download
I don't have a Fidelity NetBenefits account to test, but I would imagine the trick might work also.
Andrew Dingwall: 443 worked!!!! Thanks.
DeleteCal Learner
Hi Andrew,
DeleteThe Discover Card trick :443 did indeed work with Fidelity.
Thank you, thank you, thank you.
-Kevin N.
I can now confirm that Fidelity NetBenefits works without adding :443 to the url.
Deletehttps://nbofx.fidelity.com/netbenefits/ofx/download
Hi All,
ReplyDeleteThank you for all of your verifications re Fidelity.
The connection to Fidelity appears to be working as expected in Moneydance.
I hope that, that means this can be fixed.
Also seeing the same issue with Fidelity. -
ReplyDeleteAccess Denied
You don't have permission to access "http://ofx.fidelity.com/ftgw/OFX/clients/download" on this server.
I should note - I found this online, which suggests that Fidelity has had this issue before - so maybe they will fix it on their side. Not good that it's been a few days already though
ReplyDeletesocial.microsoft.com/Forums/en-US/1c17136e-3170-4230-8d94-b967d6a066d1/pocketsense-problem-accessing-fidelity-resolved
Dan:
ReplyDeleteIt is possible that the September 2019 thing was a transient condion of the same thing.
Changing the URL, as described by Andrew DingwallJune 13, 2020 above, should prevent this same thing from happening. These events were stressful.
Cal Learner.
So if the 443 Discover trick worked with Fidelity, maybe there's another "secret number" that will make Discover work again?
ReplyDeleteAdding :443 for the Fidelity issue worked for me. But why is that what does :443 do?
ReplyDeleteIt really shouldn't make a difference. Because data can be sent with or without the use of SSL, one way to indicate a secure connection is by the port number. HTTP, the unsecure protocol, uses port 80, but by default, HTTPS connections use TCP port 443, and all communication with OFX servers use HTTPS anyways.
ReplyDeleteExactly. 443 is not a secret number; it's the default port used for HTTPS connections. The only thing that would be affected is the "Host:" header sent by the client, so that's most likely what would account for the difference.
DeleteCal, and anyone else coming on this in the future...
ReplyDeleteThe Fidelity issue seems to have resolved itself (for me at least) by just waiting a few days. I chalk it up to their IT staff (or whoever manages their OFX stuffs) futzing around with things and breaking something without realizing it; then fixing it after a few days.
Goodness knows that happened plenty for us at Microsoft when I was working with the Money team ;).
Just curious how long my fellow pocketsense users have been using MS money? My first entry is 6/19/1999, a month after graduating from college. Who knew 21 years later it would still be a piece of software I use 3-4 times a week.
ReplyDeleteI had money way back in 94 or 95. I archived about 5 years worth of data in 2000, because it was slowing my computer down. LOL. I can't believe how many of us are still using it. I just can't find anything that forecasts like Money does. I really wish they hadn't buckled under the pressure with Quicken, Mint and Yodlee. None of them are very good at personal finances. I try a couple times a year to find new software and the amazing thing is that it always feels and looks like a downgrade. I wish they would resurrect MM.
Deletehttps://www.microsoft.com/en-us/microsoft-365/blog/2020/06/15/introducing-money-excel-easier-manage-finances/
DeleteThis ad just popped up for me. Haha. I'm not impressed. Why the heck didn't they pick up MM again and improve on it. This appears to be missing many of the great features of MM. Wonder if there is some programming in there that could help us to automate our downloads from some of the new requirements. I don't have 365, so I'm only going off the video, but it doesn't appear to have the forecasting tools. :(
My oldest dated transaction is 1986 (imported from a spreadsheet when Money came out), and pretty much every transaction involving a bank has been recorded since then. MSMoney has been very solid given that its had no maintenance for years; the pitfalls around external links and content become known and avoided without too much effort. I've dropped use of some banks when they ceased to work with pocketsense, I'm that dedicated to my MSMoney. :)
DeleteI am wondering if anyone has been able to automate this script? I tried with Task Scheduler in windows, however, with the password prompt it stalls. So, I did find where I could insert said password, but it's not secure. The next thing would be to make it so that it auto uploaded to Microsoft Money without asking me - preferably when it's already open. I need to start downloading more regularly. I just don't have the time to do it daily, like I should, right now. Any suggestions, help, thoughts are appreciated. I don't have much skill with Python. I tried to see if I could just put the password in another file, but I couldn't figure out how to either pass a variable or read and print from a .txt file. I guess it really is harder to teach an old dog new tricks.
ReplyDeleteGeekyMomJuly:
ReplyDelete"I am wondering if anyone has been able to automate this script? I tried with Task Scheduler in windows, however, with the password prompt it stalls."
Yes! I do run a script each morning well before market opens. I don't do it at 2 or 3 am, because occasionally there seems to be some delay occasionally for the OFX files to get all of the data, so I pick something around 6 am ET.
I could repair a package, but for now, let me address your password specific problem. I created a file called lines.txt. It contains just 4 blank lines (crlf). So the file is 8 bytes long.
I do stuff, including checking that this is Tu...Sat. I don't want to bring in quotes for weekends. If the money file is not open, open it. If the file I want is not open, just wait.
getdata.py < lines.txt
That is the missing trick for you. Now, whenever the script asks a question, it reads a crlf from lines.txt, and so takes the default for that question.
BUT, I don't password/encrypt my ofx_config.cfg file. If I did, I think I would need a file like lines.txt, but with the password at just the right spot. You can play with that. You can remove the encryption on ofx_config.cfg for now, and then think about going back to the encryption experimenting.
I do something else that you don't have to do. I don't have PocketSense combined OFX files. I had my thing working before that feature was added, and there was once I had a problem because one FI was not giving a good file. So I just reverted. I have my button pusher in a program called popdown2.py, and do
start popdown2.py
as a line in my batch file. No need for that if you combine.
There are other things to consider. Make sure that your scrubber gives good dates on stock quotes. Especially if the new windows scheduler runs a time that you have not normally been using. I have my own scrubber... Harder to follow, but it works. If you don't have a problem with quotes being associated with the right date, cool. No need for my scrubber.
I also write stuff to various logs in my batch file, but I seldom look at those.
Cal Learner
Robert - I've been trying for months to restore download capability for Star One Credit Union. Downloads worked well for quite some and then ceased working late last year/earlier this year. I've emailed back and forth with Star One support personal and cannot pinpoint the issue. I believe the downloads starting failing when they added two-factor authentication to their website. I have to enter my username and password and then provide an answer for a security question until I tell their website to trust my device/browser. I receive some indication in the received OFX file that this is the case because the file has a notation about "robots". Any help would be greatly appreciated.
ReplyDeleteMoneydance at http://moneydance.com/synch/moneydance/fi2004.dict shows two entries for Star One Credit Union
Delete{
"id" = "md:1919"
"uses_fi_tag" = "y"
"access_type" = "OFX"
"fi_name" = "Star One Credit Union"
"bootstrap_url" = "https://ofxdc.prd1.ncr.com/cmr/cmr.ofx"
"fi_id" = "321177968"
"app_id" = "QWIN"
"fi_org" = "DI"
"app_ver" = "2400"
}
{
"id" = "md:2781"
"uses_fi_tag" = "y"
"access_type" = "OFX"
"fi_name" = "Star One Credit Union - New"
"bootstrap_url" = "https://fs-services.1fsapi.com/eftxweb/tyfone.ofx"
"fi_id" = "94089"
"app_id" = "QWIN"
"fi_org" = "starone.org"
"app_ver" = "2400"
}
Perhaps you can try the one you aren't currently using.
The cmr one is the old one. I confirmed with Star One that the tyfone one is correct. Downloads were working fine but abruptly stopped working when two factor authentication was added to website login. Hasn't worked for me since.
DeleteAt https://www.ofxhome.com/ofxforum/viewtopic.php?id=49754, on Jan 18, fleminra suggests setting the app version to 2600 to get it to work.
DeleteTried mucking with the app version too without making any headway. I've even tried a brand new install of Pocketsense, but it fails on initial connection when trying to download accounts.
DeleteIf you are comfortable, try editing control2.py, and changing the Debug variable from False to True, by commenting the first line and uncommenting the second. Run Setup.py, option 7, to test the account. Save the program output and the appropriate ofx file in the xfr directory. Make sure to edit out any personal info such as user, password, and account numbers. Upload it somewhere and post a link. Revert the changes to control2.py.
DeleteAndrew - Here's a link to a stackoverflow forum post describing exactly what I'm seeing in response to a download request to Star One. What's shown in that initial post is exactly what my OFX reply from Star One looks like and what is captured in the Star One file in the xfr folder.
Deletehttps://stackoverflow.com/questions/50897837/python2-7-access-an-https-website-and-retrieve-content
This comment has been removed by the author.
DeleteI removed my posts regarding changing the user-agent since I don't believe it will work unless you can find user-agent what quicken is using. I played around connecting to Star One and received this response, < and > removed:
Deletehtml
head
META NAME="robots" CONTENT="noindex,nofollow"
script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3"
/script
body
/body
/html
It looks like they have the OFX server set up with Incapsula. Following the script src leads to obfuscated code, which leads to a second layer of obfuscated code.
On the stackoverflow thread, following the link to quora, it appears that this code sets a cookie, that needs to be sent with the next request. "They send you a small obfuscated JS code that sets a cookie instead of the page content for as long as you are requesting it without a cookie." Pocketsense would have to execute the javascript code in order to find the cookie.
I don't know if you can download an ofx or quicken file from Star One, but that's probably the best you can do now. Save it in the import directory, and it should import on the next pocketsense run.
Andrew - I can actually get Pocketsense to successfully download transactions from Star One by using Fiddler to go to the OFX URL and applying a POST command. Once I do that I can see the cookie settings and Pocketsense will download transactions. I suspect Fiddler is somehow sending/setting the cookies (there are two of them) which allows Pocketsense to then request the data from the OFX server.
DeleteJohn,
DeleteI think I made some progress.
1. You will need some mods to ofx.py. Make a backup copy of ofx.py first.
About Line 252, after the comment
#if both fail, revert to V1 request method
add these lines
cookie = None
if self.url == "https://fs-services.1fsapi.com/eftxweb/tyfone.ofx":
cookie = raw_input("Paste cookies\n")
h.set_debuglevel(1)
This code is where you enter your cookies. You will have to manually correct the indentation. I am turning on http debugging (verbose), so you can see what we send, and what their response is. That line can be commented out with a # if everything works.
About Line 270, In the for i in [0,1,2] loop, after the line
h.putheader('Connection', 'Keep-Alive')
add these lines
if cookie <> None:
h.putheader('Cookie', cookie)
cookie = None
This will insert your cookie into the HTTPS request. You will have to manually correct the indentation here as well.
2. Create a bookmarklet called "Show cookies" in your browser with the following as the URL (all on one line)
javascript:var x = document.createElement("P"); var t = document.createTextNode(document.cookie); x.appendChild(t); document.body.appendChild(x);
This just appends a text version of the cookies at the end of the current page. This makes it easy to copy the cookie.
3. Load https://fs-services.1fsapi.com/eftxweb/tyfone.ofx in your browser.
4. Load the "Show cookies" bookmarklet from the current page. Copy the value for
nlbi_#######=...;
Do not include the semicolon. Copy from nlbi_ up to the semicolon
For example, this is what I get in my browser when I load the page and run the bookmarklet:
incap_ses_358_1006893=hirpfmNbbVW3/VjDKt/3BM56JV8AAAAAorNjJXLh3waq3R/lgfTmaw==; nlbi_1006893=YGIgdGkXKSyvKE3QH8nh2QAAAACEUXapLjATy6stZydT/h5N; AWSALB=+1v0wSjLTi4gTZRW4nmIjbGi/ZQXq3T6B5xV81iKozEohFb2u3+OPG6ueRMhTOFC9e47+PWbMA7ZVZLvkU5AsM6e0rAZMty6sArtncKwpcxlgWAjkOdGqNuaFiYm; AWSALBCORS=+1v0wSjLTi4gTZRW4nmIjbGi/ZQXq3T6B5xV81iKozEohFb2u3+OPG6ueRMhTOFC9e47+PWbMA7ZVZLvkU5AsM6e0rAZMty6sArtncKwpcxlgWAjkOdGqNuaFiYm
I only copy this:
nlbi_1006893=YGIgdGkXKSyvKE3QH8nh2QAAAACEUXapLjATy6stZydT/h5N
up to but not including the semicolon.
The cookies may not be in the same order each time.
5. Run Setup.py and test your Star One account to see if this works. You should see a prompt to Paste cookies. Paste in the nlbi cookie, and press ENTER. If all goes well, it should work.
I don't have an account with Star One, so I tried to add an account with blank user and password, using these changes, and I am able to see an OFX response from Star One.
The response, with debugging on, has a Set-Cookie: line with this cookie:
visid_incap_1006893=8iRYVqJ9TxGlm90ZLX39BWyDJV8AAAAAQUIPAAAAAAAzZ2qOsJjxqD31wnhYkICW
This cookie is marked HttpOnly, so cannot be seen with javascript. It has a one year expiration! It is possible that on subsequent runs, you may be able to send only this cookie instead of loading the browser each time.
Andrew - I've been able to perform all of the above steps but am still, unfortunately, getting the same erroneous result when trying to download from Star One. One thing I've noticed is that the second cookie shown when running the Show Cookies applet (i.e. incap_ses_484_#######=...) doesn't match the cookie that's sent back and echoed in the debug output from ofx.py. My assumption is that the cookie passed in initially is somehow decoded to generate one or both of the other two cookies (i.e. incap_ses_484_####### and visid_incap_#######).
DeleteHere's the data I get when running Show Cookies:
nlbi_1006893=/WMrMDGK+SUOcZyYH8nh2QAAAAATaSFItfenzvbiApmuogsm; incap_ses_484_1006893=vZA7fSEAkAiXFc7OlIO3Bkr4N18AAAAACJ6WLR9MBnzu1NSplCMPig==; AWSALB=NirLaJtp/cKkN24B3+MfEyeFZ19jQRs03pCpqAYWLukjiSdqX8q50ZXKXpZ8bqxHBX7y50JElypu0fyNaX9k/T9aMygyRyNJWqQ9d4eYVJCAbLoPTnX9B5EtrxdP; AWSALBCORS=NirLaJtp/cKkN24B3+MfEyeFZ19jQRs03pCpqAYWLukjiSdqX8q50ZXKXpZ8bqxHBX7y50JElypu0fyNaX9k/T9aMygyRyNJWqQ9d4eYVJCAbLoPTnX9B5EtrxdP
Here's what I'm pasting in when ofx.py requests the cookie:
nlbi_1006893=/WMrMDGK+SUOcZyYH8nh2QAAAAATaSFItfenzvbiApmuogsm
Here's what I'm getting back from Star One:
header: Set-Cookie: visid_incap_1006893=i3WMxnhOTUSI6C0m2969Kw7+N18AAAAAQUIPAAAAAAAOkA69ILTTGUVGzyS1VC93; expires=Sun, 15 Aug 2021 14:30:02 GMT; HttpOnly; path=/; Domain=.1fsapi.com; Secure; SameSite=
None
header: Set-Cookie: incap_ses_484_1006893=nBdXdrgZdy+OFdjOlIO3Bg7+N18AAAAASoGbwsn5gZuej+psiNvD2g==; path=/; Domain=.1fsapi.com; Secure; SameSite=None
As you can see the incap_ses_484 cookies don't match.
John,
DeleteCan you try to paste in
visid_incap_1006893=i3WMxnhOTUSI6C0m2969Kw7+N18AAAAAQUIPAAAAAAAOkA69ILTTGUVGzyS1VC93
I believe that this is the long-lasting cookie since it is set to expire in one year. The cookie is marked HttpOnly so it can't be seen in javascript. If we can get this to work, we can change the sites.dat file to add an entry like
cookie: visid_incap_1006893=i3WMxnhOTUSI6C0m2969Kw7+N18AAAAAQUIPAAAAAAAOkA69ILTTGUVGzyS1VC93
to pass on the http connection for this, or any other site that needs it. This would likely work for one year before having to get a new cookie, and would not require manual intervention each time.
I believe the incap_ses_484_1006893 cookie is a session cookie that will be different each time.
DeleteAndrew - Got it to successfully work by pasting the incap_ses_484 cookie rather than the nlbi cookie.
DeleteFor Any Citi Credit Card Users - I have noticed that Citi seems to have made a change recently where they seem to be backdating the date a transaction is posted back to the date of purchase. For example if you make a purchase on July 7 and it shows pending on July 7 and July 8, then on July 9 it posts (not pending any longer) Citi will use declare the post date = July 7 (in the past it would have been July 9).
ReplyDeleteI noticed this because I am weekly downloader and have been consistently missing some transactions each week because of this. Just something to watch out for.
I too thank you. While reconciling my account, I couldn't figure out why it wouldn't zero-out, so I ran Test of citicard, which goes back 30 days, and discovered that missing transactions filled into place.
DeleteMichael:
ReplyDeleteThank you for the info on the transaction dates. I had not noticed. Now that you point that out, I see it in my register
I suggest that if you will download weekly that you set your defaultInterval to maybe 14.
In the case of CitiCards via PocketSense, they keep the same FITID for the same transaction, as everybody should. So if you download the same transaction again, Money knows to disregard. So with the longer interval, you are less likely to miss a transaction.
Cal Learner.
Downloaded the latest version, installed Python 2.7.18 for Windows x86, edited my sites.dat, and when I run SETUP.PY, after I enter my first username and password, I get "An error occurred requesting accounts from the site. please check username and password."
ReplyDeleteThis is for Chase and Citibank. With Capital one, the SETUP program abruptly quits after entering the password.
How am I getting this so wrong? Thanks!
Da Sound GuyAugust 12, 2020 said
ReplyDelete"This is for Chase and Citibank."
I presume you are talking about a creditcard, because a bank account won't work.
It has been a long time since I set up Citicard. I think that predated the new way.
What is the date on your control.py file? The reason I ask is to see if your Pocketsense files are recent.
Here is my site descriptor for CitiCard:
<site>
SiteName : CitiCard
AcctType : CCSTMT #credit card
fiorg : Citigroup
fid : 24909
url : https://mobilesoa.citi.com/CitiOFXInterface
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval:
timeOffset :
</site>
Cal Learner.
For Citibank, you must log in, go to Profile, More Settings, Manage Desktop Apps. Follow the directions before running Setup.py.
DeleteFor Chase, you must log in, bring up the three-line menu, go to Profile & settings, AccountSafe, Desktop software. Follow the directions before running Setup.py.
Thanks Cal, changing the URL helped, along with Andrew Dingwall's advice.
DeleteWhy do you say Citibank bank accounts won't work?
Andrew Dingwall: Good going. Did I know that at some point and forgot?
ReplyDeleteDa Sound GuyAugust 13, 2020 said "Why do you say Citibank bank accounts won't work?"
Nobody that I have ever seen says the bank accounts provide an OFX server for PocketSense to access in the sense that you would set up the bank account. If you find otherwise, I would like to hear it. Yes, you can get OFX files from the website. I don't classify that as working with PocketSense. But there is that new provision with PocketSense that lets you invoke a scrubber....
With the OFX via the web for the Citibank bank accounts, I also have seen a problem where the FITID is not the same on the same transaction later. That causes duplicates. I work around that by specifying specific dates for the OFX to provide. If PocketSense could actually directly get the OFX files, it would be worth making a scrubber to deal with that inconsistant FITID problem.
Note that CitiCard has had a quirk where the <ACCTID> has had the number with the first several numbers replaced with X's. The symptom is that you need to tell Money what account the file goes into. That is only a problem if it changes back and forth. I actually scrub for that.
Cal Learner
New user here. I have been able to get my Citi credit card and Fidelity and Vanguard investment accounts to work fine. My checking account is at CommunityAmerica Credit Union, a local credit union. I have turned on Quicken access through their online banking site. I can connect and the site provides my account numbers for me to select during set up. However, when I try to test it, I get an error that a required BANKID value is missing. I can't find that value in either OFX Home or Money Dance listings. I called the credit union and they don't know what I am asking for. Any suggestions on how to find the BANKID value? Thanks in advance.
ReplyDeleteTry the routing/transit number for the bank.
DeleteThat did the trick. Thanks very much.
ReplyDelete