- Added scrub routine for T.Rowe Price, to correct issues related to reinvested dividends and capital gains.
- Removed CLTCOOKIE field from request. It is not supported by Quicken, or any version of Money since 2004.
- Minor bug fix when listing accounts during Setup.
Friday, August 24, 2018
New version available
A new version of the scripts is available to download, and includes the following changes:
Subscribe to:
Post Comments (Atom)
Looks like Chase is down.
ReplyDeleteThe scripts report a Bad Gateway.
Moneydance is just giving a 2000 error.
A Quicken user reported problems there too.
-Kevin N.
Back to normal @ Chase.
Deletesomehow, "error reading http://finance.google.com/finance" kept showing up these days.
DeleteForgot to substitute curly for pointy brackets above. Here it is with the substitution:
ReplyDelete===============
I had been cutting in my own "def _scrubFidelity(ofx):".
Included in that is a time massage to prevent the downloaded prices from not being listed with a later date.
My process was to get the date from the {DTSERVER}. Remember that date only. Compute a time before that datetime of about 24.5 hours (old notes... don't remember why that number seemed good. If adjusted date was a Saturday or Sunday, back off the date to the preceeding Friday. This involved " while dt.weekday() } 4: # 0==Monday 4 == Friday..."
Change computed datetime to noon.
Then using the newly computed datetime, replace all dates with the remembered dates with the newly computed datetime.
It turns out that this substitution matched up with {DTEND}. It never matched up with the dates of transactions. I now think, due to fresh experiments, that only {DTEND} matters in Money as far as assigning dates to price quotes in the OFX file.
At some point I had switched to the released scrubber.py. A "timeOffset: -20" has no effect on {DTEND}.
I apologize for not getting on this sooner.
I suspect that a similar thing could apply to other brokers.
My simplest suggestion is to make timeOffset also adjust the {DTEND} time.
I expect to return to using my Fidelity scrubber and have it scrub times. It could be simplified I think. I can post my uncleaned version if somebody wants.
CL
timeOffset currently only adjusts {DTASOF}. I'm curious if Fidelity is sending a {DTEND} field at all? If you look at scrubber.py, you'll see a _scrubDTSTART() function, that adds a DTEND field when it's missing. It currently uses today's date, but I don't think there's a specific reason for using today's date.
DeleteI'm a Newbie to pocketsense, i have gone through your instructions, I got to the point where python queried the bank and returned a list my account numbers, however when i attempted the test I got this error "A request with this TRNUID has already been received and processed" which i found in the XFR folder. Is it possible for you to shed some light on this issue or am I doing something incorrectly that is causing this error.
ReplyDeleteThanks in advance.
I missed this comment before. That's a new error for me. Are you saying that you added the account, but when testing, got this error? It would help if you provided more details, such as the site entry, as it's likely the result of a first request failing, and then trying again with new header parameters... but the same ofx request.
DeleteI was able to add the accounts, and it appears that the connection was successful as my accounts were listed automatically. However the test returned this error
Delete"A request with this TRNUID has already been received and processed.STATUS STMTTRNRS BANKMSGSRSV "
Having the same problem with Wells Fargo. Was this ever resolved or is there a work around? sites.dat file is
DeleteSiteName : Wells Fargo Advisors
AcctType : INVSTMT
fiorg : WF
fid : 12748
url : https://ofxdc.wellsfargo.com/ofx/process.ofx
bankid :
brokerid : Wells Fargo Advisors
appid : QWIN
appver : 2600
ofxVer : 103
mininterval:
timeOffset :
Hi there:
ReplyDeleteI've been using Psense for probably about 5 years or so. A little fiddling here and there, but overall works very well, for which I am very grateful. I currently have it reliably running (the Dec '17 version)on a Windows 7 Home Premium Lenovo laptop. Mainly Vanguard, and one 401k at Fidelity.
Yesterday I went to transfer this operation to a newer Lenovo laptop running Windows 7 Pro SP1 (both are 64b versions). I have struggled with Fidelity NetBenefits mightily and currently can't get it to work. I first loaded the same Python version as on the older laptop: ActiveState 2.7.13.2716. Then the newest Psense from 24Aug18 (subject of this Thread), then the MM Sunset. After I set my sites.dat the same as my old version (after running setup.py to make it new), I set up the accounts. Summary is: Vanguard works fine, but Fidelity returns this after entering "y" in response to "Do you want to test...new account now (y/n)?" :
>>>>>>>>>>>>>>>>>>>>>>
** An ERROR occurred sending POST request to Nbofx.fidelity.com
Exception type :
Execption val : [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed <_ssl.c:661>
Traceback :
File "C:\Users\me....\where my Psense files are\Setup.py", line284, in
config_account<>
File "C:\Users\me\where my Psense files are\Setup.py", line109, in config_account
with open as f: response = f.read<>
IOError: [Errno 2] No such file or directory: 'acctQuery.tmp'
<<<<<<<<<<<<<<<<<<<<<
The above was very hard to capture, as in the newest Psense version when it fails, the Python window closes just after the result is echo'd out. In the older Dec17 version failure mode, the window stays open.
I have tried:
deleting and removing all Python, Psense and MM Sunset from new LT, reloading all with new from scratch sites.dat and same result,
then, loaded the old (Dec17) Psense scripts in a new folder and set up using that with a new restored MM Sunset. Result is basically the same, without the lines starting with Traceback...
So, two computers, same OS, same MMSunset, same Python, same Psense; different result. Arrrrghhh!
I feel I must be missing something simple here, but just can't figure it. Verified old laptop still downloads fine from Fidelity, so it's not them. Selected 'modify account' on old laptop for Fidelity account and it tested fine. FYI, I have installed Python using the Master admin account on the new laptop, then the Psense and Money in my User account. Don't know if that would affect Python behavior, I'm definitely not a coder, just a moderate to high competent computer user.
Any insights, suggestions, or request for clarifying info is highly desired; I am immensely frustrated at the moment. I left out probably 3 or 4 long delete, remove, reload, retry scenarios all done to no avail.
Thanks,
Hi Joryde,
DeleteI am able to connect to my Fidelity 401k account using the 180824 version of PocketSense OFX scripts. I recall running into SSL problems with several other investment accounts back in July or August this year and believe it had to do with them disallowing all SSL versions [1.0, 2.0, and 3.0] as well as TLS versions 1.1 or lower due to security risks. I upgraded my copy of python to version 2.7.15 which provides TLS 1.2 support and my PocketSense OFX connections started working again.
For reference, here is my sites.dat entry for Fidelity 401k.com:
SiteName : Fidelity NetBenefits
AcctType : INVSTMT #investment
fiorg : nbofx.fidelity.com
fid : 8288
url : https://nbofx.fidelity.com/netbenefits/ofx/download
bankid :
brokerid : nbofx.fidelity.com
ofxVer :
appid :
appver :
mininterval:
timeOffset :
Cheers, David Casada
Joryde:
ReplyDeleteMake sure that you have done successfully done Window Updates.
Consider using a command window for early testing:
open a command window with cmd
cd "C:\Users\me....\where my Psense files are"
python setup.py
My suspicion is that there could be some kind of path problem. Using the command window might get around that. But even if not, it should get around the problem of error messages going away on you.
If still no joy, for troubleshooting consider starting the command window by right-clicking the launch icon, and choosing RunAsAdministrator.
Just for the heck of it, here is my Fidelity descriptor in sites.dat
I presume yours is pretty much the same, but I might as well post it (with square brackets replacing pointy brackets for posting):
[site]
SiteName : FIDELITY
AcctType : INVSTMT
fiorg : fidelity.com
url : https://ofx.fidelity.com/ftgw/OFX/clients/download
fid : 7776
bankid :
brokerid : fidelity.com
appid :
appver :
mininterval :
timeOffset:
[/site]
Hi Joryde: Is there a reason for setting up the scripts from scratch on the new install? I'd just copy the following files to the new installation from the old:
ReplyDelete1. sites.dat
2. ofx_config.cfg
3. connect.key
Of these, the connect.key file *may* be the culprit, if Fidelity requires client-specific authentication. I'm pretty sure that they do, based on some recent user input. It's possible to get a different computer "authenticated" from scratch, but it may require logging into their site, or contacting them in some way. I don't use Fidelity, so not sure of the exact steps on that option.
I am pretty sure Fidelity does not use a connect key. I just tried a test using a folder holding an older PS version with no connect key, and the test succeeded.
DeleteWhen I saw the "Execption val : [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed [_ssl.c:661]" I thought maybe a Windows update would get updated certificates or something.
CL
Fidelity update...
ReplyDeleteI read the comment in the last post about Fidelity using their new "Fidelity Access" to control 3rd party access to their accounts. In dropped a line to customer support and I got the following response:
"We will continue to provide access to OFX downloads, and access will not be effected by the Fidelity Access feature. We hope this message addresses your concerns. If you have any additional questions, please contact us at your earliest convenience."
Hopefully this is true, and we are not affected by their changes.
Carlos, thanks for that.
DeleteCL
Some updates:
ReplyDeleteFirst, prob not relevant, but I tried both unencrypted and encrypted and get same result.
Windows updates are nearly same, both PC's waiting for Aug versions of Defender, and Malicious s/w removal tools..., but basically up to date.
This will show my level of knowledge on Python; I started from scratch because I was going to use the latest PS, and I wasn't sure what might have changed in any of the files that come in the zip, and if they have any relationship with each other. Caution on my part, unnecessarily I guess...
So now I just looked at my PS folder and renamed the three files Robert mentioned above to xxx.ytdy, then was about to transfer over the three from my older PC when lo and behold I noticed I don't have a file called "connect.key" in my new Psense folder!
I went ahead and copied over just those three from the old PC to the new and tried it, and of course got the same failure. Not really surprised by that, but it does seem like I must need the connect.key file.
I went through the instructions like I was a newbie, following carefully. Why would the connect.key file not be created? The new PC uses the basic Win7 encryption, EFS(?) or something like that. The old one uses nothing; could that be a factor?
File encryption doesn't matter. The reason for no connect.key file is that the site entry is configured to use ofxVer 102 (default, not specified). Connection keys are only used when ofxVer=103 or higher. Since you're not using ver 103 on the first computer, it shouldn't be needed. The scripts do not manage encryption certificates directly, but rely 100% on the Windows "certificate authority" via the python ssl library.
DeleteAs a next step, I'd recommend uninstalling python and reinstalling the latest 2.7.14 release.
Joryde, here is a variation on Robert's suggestion.
DeleteCopy your entire PocketSense folder over to something inside your Documents folder. Make sure that folder has a different name (such as PS2) than the one you have been using. Then in a command window, CD over to that folder and run that copy of setup.py.
You could switch back and forth and experiment independently.
CL
Here is my sites.dat description for Fidelity on old computer (same for new)(It's a 401k, so uses Fidelity NetBenefits site, not their regular investment site)
ReplyDelete(site)
SiteName : Fidelity NB
AcctType : INVSTMT
fiorg : nbofx.fidelity.com
fid : 8288
url : https://nbofx.fidelity.com/netbenefits/ofx/download
bankid :
brokerid :nbofx.fidelity.com
ofxVer :
appid :
appver :
mininterval:14
timeOffset :
(/site)
There is a connect.key in my currently working (in old PC: 1st) with a date of 6/9/18; it's the only file with that date in my 'PsenseActive' folder. I think I went from around April to early June not using encryption in setup.py for some reason, but then reset it on that date. There is data in it. Other than this FidelityNB site, the rest of accounts are Vanguard.
CL: Did you mean copy the PS folder that works on my old machine?, or the one that's not working on the new. I did make the folder structure the same (where PS is hierarchically, and name) but not change.
Robert: I'll try the Python update tomorrow, no time today.
Thanks again!
Jon
"CL: Did you mean copy the PS folder that works on my old machine?, or the one that's not working on the new. I did make the folder structure the same (where PS is hierarchically, and name) but not change."
ReplyDeleteThe working one. The twist was to have two separate folders (including the old working one), and further to put the copied folder into your Documents folder to eliminate any potential permissions problems.
CL
Hello,
ReplyDeleteUninstalled ActivePython 2.7.13,
checked registry, found few remnants of python, left alone, closed regedit,
Rebooted,
Checked registry, no python remnants,
In master account (admin) downloaded and installed (run as administrator) ActivePython 2.7.14
Tried setup.py and same (NG) result.
Point: only FidelityNB fails, I checked Vanguard using the same sites.dat, and no issues, downloaded successfully (same as before).
Directory for PS is in MyDocuments/MMDs in user directory. Then moved it to directly under MD (My Documents\PocketSense Active and same result.
I also used the ofxget java applet on the ofxhome website for Fidelity NetBenefits and downloaded a file successfully. I don't know if anything in that file would be helpful; Don't think I want to post it in entirety.
Thanks again,
Jon
I just stumbled across this:
ReplyDeleteFrom Robert's forum with the release of 10 November 2017, waaaay down at the bottom there's this:
>>>>>>>>>>>>>>>>>>
AnonymousMay 11, 2018 at 2:13 PM
I was getting an ssl error with fidelity brokerage
Message was
** An ERROR occurred sending POST request to ofx.fidelity.com
Exception type :
Exception val : [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
To get around it I had to tell the https connection to trust unverified certificates with context=ssl.create_default_context() in ofx.py. This was done due to a change in python 2.7.9 where it now fully validates certificates.
https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
I'm not sure I know what to do with this info; can I modify ofx.py??
Joyrde: I remembered the issue you found, but since it self resolved, I let it go. Now that you've tried obvious options, I may revisit adding a site option to force "trust certificate". I was (and am) hesitant, as a bank ssl certificate should be validated by the windows CA. What's happening, for reasons I don't know, is that your windows CA isn't fully trusting the bank certificate. I can add the option to force trust, but it's important to understand that doing so bypasses the windows validation. Its not such a bad issue for this case, as you can check the ofx response and verify that it's as expected.
Delete- Robert
Hi Robert: I think CA means certificate authority? Since I'm having no issues with Python, PS or Money on my old PC, I think it must be related to the Win7 differences. It occurred to me yesterday. When I started bringing up my new PC, I made a full admin account called Master, and a User account to generally run things in; this based on advice that that's a more secure setup overall. It occurred to me yesterday, that I should try installing Money and PS in that Master account and see if that fixes it. Gonna try that a little later today.
DeleteThanks,
Jon
Well, tried running from admin account and no change, but I guess that was expected. Don't understand the whole CA thing, and where/if there is something else I can do to change the OS behavior on PC end. I'll research that end more now.
DeleteI put https://nbofx.fidelity.com/netbenefits/ofx/download into a browser and checked the certificate. The certificate is issued to ofx.fidelity.com.
ReplyDeleteYou could try playing with the URL in sites.dat. It may well not help anything. I don't know if Python would be upset by the mismatch.
I did a little searching around to try to learn some more about certificates and authorities. Head spinning, and I'm not that sure I get who's the issuer and who's the authorizer in the transaction I'm trying to get python to execute, but I think the bank server offers a certificate, and if the cert authority on my PC says it's good, things progress. Is that about right?
ReplyDeleteI did find certmgr.msc and explored and compared the lists of certificate authorities between the two Win7 PCs. The old one has a few more in some of the lists. Also, at OFX Home site, under the Validation box for Fidelity NetBenefits listing it shows: SSL -- Passed by Entrust, Inc. In both the 'trusted root' and 'Third-Party' certificates listings, there are three Entrust entries in the old PC (the one that works with NBofx.com) and one entry in the new PC.
I also understand I can export and import these certificate authorities.
Is this something I should try?
...a bit later:
As Anonymous mentions above, I also put https://nbofx.fidelity.com/netbenefits/ofx/download in my Firefox browser and finally figured out how to view the certificate. In the Certificate Viewer>Details tab it lists "Entrust Root Certification Authority - G2" as top of hierarchy, then "Entrust Certification Authority - L1M", then "ofx.fidelity.com" (all w/o quotes of course). The one ending in xxx - G2 is one of the two listings that is missing in the new PC. Should I try to export/import this cert auth? I'd like some confirmation that I won't screw anything up because this is new territory for me.
Thanks,
Jon
Some progress, some retrograde...
ReplyDeleteAfter researching CA's, and trying to figure out how they get in your CA list (as shown by certmgr.msc) I decided to try logging in to Fidelity and downloading through their website. My default browser is Firefox, and a saw no change after that. Some days later, I decided to try the same action using Internet Explorer (IE11 in my case). Lo and behold a new Entrust CA showed up in certmgr.msc!! I was still not getting setup.py to stay open when I double-clicked from my user account but after trying it from the command line I got a 'access denied', which explained why the window wouldn't stay open (or setup.py run). This told me the Psense setup issue was just access control on the PC. Restored the PC to before I started working on this, and then installed Python in my 'standard' user account by changing in installation setup. After this, with Psense files in same user account, ran setup.py and it worked! Was able to setup my FidelityNB account, and download and import into Money successfully!!. Yea.
So then I thought I'd wrap it up by setting up my Vanguard accounts and be done with it. Now Vanguard won't work. When I run setup, it says there's an error retrieving my account numbers (worked fine when I first tried it couple weeks ago) and to check user name and password, but I get the option to continue with setup so I do... I then get a notice of invalid ofx received and to check it. Open it in Excel and I get three blocks of text (nothing like an ofx, seems like a generic website reply or something:
first: Warning, Vanguard.com will not work properly with JavaScript disabled!
second: We're sorry. The page you requested could not be found. If you used a bookmark or a favorite place to access this page, the link is no longer effective. Please go to the Vanguard homepage and navigate to the desired page.
Third: If you continue to experience difficulty accessing pages on our website, you may need to delete your temporary Internet files or adjust your browser settings. Please visit our Browser FAQs to learn more about the software and settings that work best with our website.
Not very helpful to me, but it at least doesn't seem like a CA thing. Checked my sites info, and it's the same as it's always been. I don't seem to have Java on either my old or this new machine, so I don't get the first part at all.
Any ideas on this new wrinkle?
Thanks,
Joryde
Jorde, I am glad to read PocketSense downloads are working with Fidelity NetBenefits for you now.
ReplyDeleteIn your second paragraph, I presume you are refering to a *.ofx file in your XFR folder. When viewing that file, I would try a simple text editor, such as Notepad.
Your third paragraph seems to be about failing to be able to log in with IE11.
See https://support.microsoft.com/en-us/help/3135465/how-to-enable-javascript-in-windows to see if that helps. https://www.java.com/en/download/help/ie_tips.xml is another URL that might offer some help. This is the URL I use to log into Vanguard: https://investor.vanguard.com/home
CL
Correct on 2nd paragraph; looked again using notepad; lots of code, some html, definitely not an ofx format (doesn't look anything like the Fidelity ofx).
ReplyDeleteThe third paragraph is actually a continuation of the 2nd; it's just the three blocks of text that show up when the site response (ofx) is imported into Excel (I can find the same text in the notepad version).
I have no problem logging in to Vanguard through either FF or IE browsers on this PC. What's wonky is when I try to set up Vanguard accounts through Psense setup.py. Seems to connect, but can't retrieve account numbers, and then returns this strange file instead of ofx. I could post the notepad version, but it's a ton of text (16kB worth!).
Thanks,
Joryde
Glad you figured out the certificate issue. I was going to add a 'forceTrust" option, and still may... but it shouldn't be needed for a valid ssl connection. I use vanguard myself, and just tested, with no issue. My sites.dat entry is below. Login to vanguard via their website on the same computer, and verify that it works that way too. Also, Vanguard may support ofxVer=102, but I've been using 103 for a long time.
DeleteSiteName : VANGUARD
AcctType : INVSTMT
fiorg : Vanguard
fid : 15103
url : https://vesnc.vanguard.com/us/OfxDirectConnectServlet
bankid :
brokerid : vanguard.com
ofxVer : 103
appid :
appver :
mininterval:
This is my sites.dat descriptor for Vanguard (fund only):
ReplyDelete===============
SiteName : VANGUARD
AcctType : INVSTMT #investment
fiorg : Vanguard
fid : 1358
url : https://vesnc.vanguard.com/us/OfxDirectConnectServlet
bankid :
brokerid : vanguard.com
appid :
appver :
mininterval: 365 #Vanguard doesn't like short intervals? Use 30 day wind!
timeOffset : #-9 offset helps some *if* no scrubber.
===============
CL
http://www.ofxhome.com/index.php/institution/view/722 shows they changed FID from 1358 to 15103. I intend to keep my existing number as long as it works.
ReplyDeleteMy comment on timeOffset probably has no validity. Being a comment, of course it does not matter.
CL
Robert,
ReplyDeleteI am able to login to Vanguard using either browser (IE or FF) no problem. I'm still using the 1358 fid as is CL. If I tried your sites config with ofx ver = 103 does that mean I would get a connect.key file? BTW, I typed in the server address in the url line of sites.dat, into my IE browser and got the same text page paragraphs I received in the "OFX" file that came back from the setup.py attempt.
CL,
I see on ofxhome.org they changed fid on 11 March 2018. My old PC still works fine with 1358. Anyway, I changed to 15103 but still get same result.
Re-summary:
run setup.py > get Main Menu > Selection: 1. Add or ... > get Site List > Enter Site#: 2. VANGUARD > Configure account for VANGUARD
User name: type and enter
Account password: type and enter,
get:
An error occurred requesting accounts from the site. Please check username and password.
Continue configuring account (Yes/No):
Yesterday, I selected yes immediately above, and went ahead and entered account numbers. Today when I do this:
setup.py > get Main Menu > select 2. List Accounts > Show account connection keys? Y/N > enter "Y" I get the Fidelity account and two vanguard accounts including account numbers and connect keys. I suppose this info is in one of the other files in my Psense folder, so is not much meaning maybe.
Just entered url: https://vesnc.vanguard... into IE browser on old PC (one that works) and I got a different response. This one says "The page you're trying to reach is currently unavailale. THis may be a temporary error, so please try again. If the problem persists, contact us." (same cert path though).
No idea if that means anything or not.
Thanks,
Joryde
Joryie said "Just entered url: https://vesnc.vanguard... into IE browser on old PC (one that works) and I got a different response. This one says "The page you're trying to reach is currently unavailale. THis may be a temporary error, so please try again. If the problem persists, contact us." "
ReplyDeleteMe too. On the one that does not work, make sure in InternetOptions->Advanced that "Use TLS 1.2" is ticked
CL
OK, both use TLS 1.2. Here are the schemes, and I don't think that there would be a setting making Fidelity work but not Vanguard:
ReplyDeleteFidelity:
Connection Encrypted (TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 256 bit keys, TLS 1.2)
Vanguard:
Connection Encrypted (TLSFRSA_WITH_AES_12_8_CBC_SHA, 128 bit keys, TLS 1.2)
CL
Hey there; sorry for taking so long to come back to this but I finally found a solution!
DeleteDecided to look at Python install. Did a search of 'C' for "Python" and got many hits outside of the install I knew was there! Back in the summer I had installed GIMP, the open source graphics editor and it had an install of Python come along with it (V2.7). I uninstalled GIMP, deleted USERS\me\ASPython and also uninstalled a program called Darktable (for photo work) which I had tried just for experimenting but have no plans to use, and also uninstalled Intel WiDi, which also has a Python core evidently, and is some outdated wireless monitor thing that never took off I guess.
After all that I searched the registry for Python and got nothing so re-installed Python, Money, etc. and voila; everything Pocketsense worked fine. As usual, with hindsight I guess I should have looked for another Python before all the certs experimenting, etc., but things are working pretty good now. Thanks for all the help!!
PS: I would like to re-install GIMP, but not sure how to keep it from installing another Python version. Will make a restore point, try it I guess and see if it sees the one that's there already.
Just fyi... I have GIMP installed myself, and tried Darktable for a bit w/ no issues, so likely something different.
DeleteDid you install GIMP after ActiveSense Python? I surmise that since I already had a Python core installed and put one on top of it, things went haywire. Clearly, the only thing I did to get Pocketsense working was to uninstall GIMP and the associated Python and then re-install ActiveSense Python, Money, etc.
DeleteI am intending to re-install GIMP someday, and was hoping it would be obvious how to avoid installing another version of Python. I thought either your instructions, or the ActiveSense page instructions warned against multiple installations of Python; seems like I saw that somewhere...
Anyway, I'm glad to hear it can work!
I just updated to the newest PS version and as account updates cycles through all my accounts I'm now getting the following message after some, but not all, of the account OFX downloads: "+Scrubber: Null time values updated." It doesn't seem to affect anything and the combined OFX file seems to be processing correctly, but I'm curious what it means and if I'm missing something? Anyone else seeing this message when the Update script executes?
ReplyDeleteHi Ken,
DeleteDo you have quietScrub in sites.dat set to Yes?
I don't receive the Null time... message but I DO have quiteScrub set to Yes.
-Kevin N.
@Kevin. Thank you! quietScrub was set to No. I changed it and all the messages have disappeared. Problem solved!
DeleteHi Ken,
DeleteI'm glad to hear that it worked. Thank you for reporting back with your results.
-Kevin N.
I will quote from the OFX spec: "Note: Developers should consider the possibility of a date change due to timezone conversion.
ReplyDeleteA datetime value in the GMT timezone with a time of 12:00:00 (noon) would be converted to
another time on the same date in every timezone. For example, 199812251200 remains
Christmas Day in every timezone."
So that fix keeps the transactions from being a day earlier than they should be.
CL
Also, correction from a string I posted above. It should have been
Connection Encrypted (TLS_RSA_WITH_AES_128_CBC_SHA, 128 bit keys, TLS 1.2)
not that it mattered.
@Anonymous. I don't know if your comment was in response to my question. If so, I should have been more specific. The "+Scrubber: Null time values updated." message I'm getting doesn't appear consistently. For instance, I have a couple different credit cards from the same bank yet one will get the message and the other doesn't. I guess I'm trying to understand (1) is this a message that's normal with the most recent PS update and (2) why doesn't it consistently show up for the same accounts?
DeleteYes, Ken. I had gone into a longer writeup but had lost that. So I mainly just snipped what the spec says.
ReplyDeleteAnyway, where the null time comes in is that some programmers only care about the date and want to ignore the time. So they put zeros into the time fields. Unfortunately zero corresponds to midnight. So if the time zone named in the file is to the east of your own timezone, Money translates that midnight eastern time to 11 pm or so the day before. Anyway, the bottom line is that null refers to a zero time. I think PocketSense figures if they did not put zero, they might have really meant it.
CL
Robert,
ReplyDeleteI'm new to creating a script for downloading financial institution transaction data. Creating the site.dat file seems straightforward, but running setup.py leads me to a dead end. Are there instructions for running setup? Somehow my setup is not getting me to an interaction where I can enter my site account number, username and password. So far it's not intuitively obvious to me. Here is my site data.
SiteName : Principal Funds for Dan
AcctType : INVSTMT
fiorg : principal
fid : 16857
url : https://ofx3.financialtrans.com/tf/OFXServer?tx=OFXController&cz=702110804131918&cl=722120617142015
bankid :
brokerid :
ofxVer :
appid : QWIN
appver : 2400
mininterval:
timeOffset :
It's the only site I want to query at this time.
Dan D
If you run Setup.py, and add an account (option 1), what specifically happens?
DeleteI'm not sure what was happening, but after a bit Setup started responding to my Sites.dat file and taking info on user name and password. So far so good.
ReplyDeleteHowever when getting to testing the site, Setup accurately finds my 3 fund investment "accounts",which consists of a "fund number" and overall "user account number". So at Principal it's organized as "fund number"-"account number". When I test this setup, it connects OK but then fails out with the error "ACCOUNT NOT FOUND"
If I could figure out how to get around this blog post error "Your HTML cannot be accepted: Tag is not allowed: CODE" I could show you my sites.dat data and the OFX return text.
ReplyDeleteDaniel Dunatchik, you could put what files you want into a zip file, and upload to http://tinyupload.com/ Then post the URL here.
ReplyDeleteAlternatively, replace all pointy brackets (less-than or greater-than symbols) with square brackets, and post directly here.
Here is my latest sites.dat file for site Principal:
ReplyDelete[site]
SiteName : Principal
AcctType : INVSTMT
fiorg : principal
fid : 16857
fi_name : Principal Funds
url : https://ofx3.financialtrans.com/tf/OFXServer?tx=OFXController&cz=702110804131918&cl=722120617142015
bankid :
brokerid :
ofxVer :
appid :
appver :
mininterval:
timeOffset :
[/site]
Here is the failing OFX output file resulting from Setup testing of the site:
OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
[OFX]
[SIGNONMSGSRSV1]
[SONRS]
[STATUS]
[CODE]0
[SEVERITY]INFO
[MESSAGE]SUCCESS
[/STATUS]
[DTSERVER]20180921163514.208[-5:CDT]
[LANGUAGE]ENG
[DTPROFUP]20180921163514.208[-5:CDT]
[DTACCTUP]20180921163514.208[-5:CDT]
[FI]
[ORG]principal
[FID]16857
[/FI]
[/SONRS]
[/SIGNONMSGSRSV1]
[INVSTMTMSGSRSV1]
[INVSTMTTRNRS]
[TRNUID]307c39c1-fad1-4542-9e14-1faaed3cafec
[STATUS]
[CODE]2003
[SEVERITY]ERROR
[MESSAGE]ACCOUNT NOT FOUND
[/STATUS]
[/INVSTMTTRNRS]
[/INVSTMTMSGSRSV1]
[/OFX]
It is not obvious why Setup can accurately find my account number and account funds but the response at the site is that they are not there, I would guess.
Dan D
Dan: If I read the first part right, the account list is being returned as "fund number - account number". If so, try manually entering *only the account number* at the prompt, rather than selecting the line #.
DeleteI am not sure how to enter there account number manually
ReplyDeleteNevermind, I see what you're saying
ReplyDeleteRobert, to answer your question the account list is being represented in SETUP as "fund number,account number", a comma delineating not with a hyphen, and the fund numbers are accurately represented. However, the account number's leading 0 is suppressed. I don't know which process had suppressed the leading 0, but I do know it's critical to a successful return. If I use just the account number (with leading 0 included) and no representation of the fund number, I get a successful return with no errors, but also no transaction data. If I append the fund number either in front of or trailing the account number, with either a hyphen, comma, or space between them, I get a return with error stating ACCOUNT NOT FOUND. If I append the fund number directly to the account number I also get a successful return, however no data is represented either.
ReplyDeleteI am baffled with the results. I do know that Principal Funds has always had a peculiar representation of funds within a user's account. If I go to the Principal site and view my account, I can only view and/or download history on one fund at a time, inconvenient, but workable. They used to have a working OFX download generator, but recently it has stopped working. That forced me to download CSV data which I've had to manually enter into Money, and was my incentive to try using PS for investment transaction tracking. I've been using PS for years to import price data.
At this point I'm not sure what to try next. It's proven we can get into the site and it recognizes my user credentials. However there is some protocol that is necessary to get it to feed transactions on a particular fund.
Maybe one of your followers has more experience with Principal Funds that he/she would be willing to share?
To add to CAl's response: when you say "no transaction data", what info is in the ofx file?
DeleteDaniel:
ReplyDeleteTry adding mininterval: 365 to your sites.dat descriptor for Principal.
Did Money say it found an investment during processing and ask if you already had defined such an investment, or did Money recognize having received the content at all? Even if there were no transactions, Money could get a price or some other info from the OFX file. If you did not feed the result to Money, try that.
CL
Anonymous,
ReplyDeleteI didn't specify a minimum interval because setup appears to be querying an interval of 30 days by default, which is what I want. I went ahead and put in a specification of 30 in the sites.dat file, as that is what is typically needed for me. A specification of 365, if the query worked, would produce many duplicate transactions. The specification of 30 guarantees that there should be transaction data in the interval.
I didn't send the results to Money but examined the OFX file for error messages. In what I called failing queries, Setup discovered error messages during testing and reported it. In the few query attempts that produced no errors, I tried to import them into Money and it reported that the import file appears to be invalid or contains corrupt data. So those were no good either.
Daniel
** Not directly related to PocketSense ** [I hope I'm forgiven]
ReplyDeleteTarget recently re-did their entire Red Card web site and in all their wisdom eliminated the ability to download OFX files. They still permit QFX downloads, but the QFX won't load into money. The old "corrupt data" message.
Anyone else use Red Card and willing to help me debug? I found one problem right a way - space after the header colons. I know Money doesn't like that. There are also a number of additional fields compared to other OFX files, but I haven't narrowed down where the issue may lie.
Quick update, in case anyone else runs into these issues. Recap: After downloading a QFX from Target RedCard I rename it to an OFX and try to ingest, but Money chokes. Using the OFXAnalyzer tool I identified two issues: 1) The header lines have a space after the colon, 2) the file has the following unsupported tags: CORRECTFITID, REFNUM, SIC
DeleteI made some minor modifications to the PocketSense scrubber to handle both of these issues. You still need to download and change the extension of the file manually, but then you can let the scrubber clean up the file.
I'm working on some proposed changes to PocketSense (I will submit to Robert when ready) that would allow PocketSense to fetch already downloaded files from a specific directory, scrub them, and add them to the combined.ofx file.
Carlos: have you seen the hleofxquotes fixWellsFargo.py program? Since it rebuilds the header, it may be directly usable for the malformed header of Target.
DeleteCL
Robert, I just want to make you aware of another issue at https://microsoftmoneyoffline.wordpress.com/2018/09/04/fixing-wells-fargo-qfx/
ReplyDelete-ameridan
Interesting. I'm curious if it's really missing terminators, or just the wrong codes for Windows. Anyone have a raw ofx/qfx file from Wells Fargo, with personal info removed, to share? If so, send to pocketsense1 at gmail dot com.
DeleteI think the Wells Fargo bank problem is web download and not via an OFX/QFX server.
DeleteCL
Thanks CL, I wasn't aware of that. Regardless, I wanted Robert to know of the issue, in case he found it useful.
DeleteHi Robert,
ReplyDeleteI see that T. Rowe Price just paid out dividends on some of its mutual funds on 180926, and the PocketSense OFX scripts scrubbed them perfectly for import to Money. Just letting you know the recent update seems to work and thanking you for your support.
Cheers, David Casada
Robert, Here is an example of the OFX file that Principal produced from PS query where I input only the base account number and no representation of the fund number. No errors are noted and the file imports into Money without errors and without transaction data either, as expected.
ReplyDeleteOFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
[OFX]
[SIGNONMSGSRSV1]
[SONRS]
[STATUS]
[CODE]0
[SEVERITY]INFO
[MESSAGE]SUCCESS
[/STATUS]
[DTSERVER]20180924143730.907[-5:CDT]
[LANGUAGE]ENG
[DTPROFUP]20180924143730.907[-5:CDT]
[DTACCTUP]20180924143730.907[-5:CDT]
[FI]
[ORG]principal
[FID]16857
[/FI]
[/SONRS]
[/SIGNONMSGSRSV1]
[INVSTMTMSGSRSV1]
[/INVSTMTMSGSRSV1]
[/OFX]
Any attempt to query with a representation of the fund number and account number concatenated produces errors in the OFX file and also does not import into Money.
Dan D
Dan D, are any of the OFX files, that make errors, from XFR folder bigger in size than the one you posted above?
DeleteIf so, how about posting one of those after removing the account number and making the bracket substitutions. You can use curly brackets {} instead of square if you prefer. That would be better because square brackets are used in the actual OFX files.
CL
Anonymous,
ReplyDeleteBelow is the results of a failing query where I have represented the account number as "fund number,account number" as represented in setup. Any similar representation produces the same result in the OFX file. Note the following result output contains no account numbers.
OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
{OFX}
{SIGNONMSGSRSV1}
{SONRS}
{STATUS}
{CODE}0
{SEVERITY}INFO
{MESSAGE}SUCCESS
{/STATUS}
{DTSERVER}20181002160656.663[-5:CDT]
{LANGUAGE}ENG
{DTPROFUP}20181002160656.663[-5:CDT]
{DTACCTUP}20181002160656.663[-5:CDT]
{FI}
{ORG}principal
{FID}16857
{/FI}
{/SONRS}
{/SIGNONMSGSRSV1}
{INVSTMTMSGSRSV1}
{INVSTMTTRNRS}
{TRNUID}cd147eb3-5ca1-4c1c-94a5-e03326b4d962
{STATUS}
{CODE}2003
{SEVERITY}ERROR
{MESSAGE}ACCOUNT NOT FOUND
{/STATUS}
{/INVSTMTTRNRS}
{/INVSTMTMSGSRSV1}
{/OFX}
Importing this file into Money produced no errors on import but also did nothing to my account holdings.
I think my next step would be to try to get some insight from Principal customer support. I'm not sure how many customers are actually trying to output OFX from them anymore.
Dan D
Robert, Anonymous
ReplyDeleteI have gotten a good OFX file from Principal Funds by downloading a transaction history from my account web site. Up until I tried it today, it has not been producing good results for me. This file imports into Money and contains accurate, up to date transactions, that I know are valid. I scrubbed the file of my account number information but left the pointy brackets as is. You can download the zip file at this location https://app.box.com/s/u10ig4hubr1zu03zblusk571hf8rmm2k
Note, the ACCTID is represented in the OFX as "FundNumber,AccountNumber", exactly as setup represents them when it checks it for the first time with my user credentials.
I am suspicious the the problem is the OFX version that I am calling on in sites.dat (default) They seem to be using OFX version 200. Does that look right? I will try PS again with the OFX version in sites.dat. If you spot any other OFX issues I should have accommodated, I would appreciated your feedback.
Daniel, I applaud your perseverance. Several of us, including myself, on the Ameridan PocketSense site were trying to figure out how to get downloads from Merrill Edge accounts to download using PS. We finally gave up and decided to be satisfied with the OFX option on their website. Best of luck getting this to work. Sometimes, though, it just isn't possible to figure out what combination of parameters will finally work.
ReplyDeleteBTW, if someone here has figured out how to use PS for Merrill Edge please let me know how 😊
Daniel. I tried your file. I had to reduce the size of the ACCTID because that field is limited to 22 characters. It passed the parser once I changed that to "FundNumber,AccountNum".
ReplyDeleteThe file that works is indeed OFX 2.x You could try telling PocketSense to use OFX 2. I think to do that you would use
ofxVer:211
in the sites.dat descriptor rather than leaving ofxVer: blank as you have it now.
C.L.
Yes, I agree. I will be trying this with OFXVer specified,
ReplyDeleteIn trying to download my data from Chase, I received the following error:
ReplyDelete** An ERROR occurred retrieving POST response from ofx.chase.com
Exception type :
Exception val : [Errno 10054] An existing connection was forcibly closed by the remote host
I've had zero issues up until the past couple of days. Any ideas?
This comment has been removed by the author.
DeleteI've gotten this today, too. I don't think it's a Pocketsense problem, since pinging ofx.case.com gets timeouts, too. I sent their Twitter team a message and got a response that said essentially nothing.
DeleteSaw this in the MoneyDance forum: http://help.infinitekind.com/discussions/problems/48352-no-longer-connects-to-financial-institution-online-chase
DeleteI'm hoping that this was something that Robert added quite a while ago and has worked well since then. I checked Chase's web site, and I don't see anything it wants me to do. It is also weird that we can't successfully ping ofx.chase.com, although they may have changed the site name.
DeleteI called Chase. They said that they were having trouble with Quicken, and that they thought it would take one or two business days to fix it. That seems quite incredible (in a bad way!) to me, but such is life, I guess.
DeleteI just ran PS update for the first time today and both my Chase credit cards updated without any problem. As I said a couple days ago, these things happen and then get fixed within a few days. It's rare that something is so significantly changed that PS stops working for these accounts and a programming update is necessary for PS.
ReplyDeleteJust tried this morning. Everything's back to normal.
DeleteDitto
DeleteRobert has solved the issue of some Schwab downloads not working, although at present I may be the only one. It turns out that the Zero-Dollar transaction scrubber doesn't always do a thorough job for Investment accounts, like it does for Credit Card transactions (which is why it was written in the first place), so until he publishes an update, if you are having issues with Schwab, disable the "Remove $0.00 transactions" option in your sites.dat
ReplyDeleteLongtime user of Pocketsense. I have been having a problem downloading transactions from Chase Credit Cards.
ReplyDelete- We have two different Chase credit cards.
- Both were working fine until couple of weeks ago.
- One card continues to work without issues.
- The problem started about couple of weeks ago with "socket error." I did not pay attention or look into the OFX file for the actual error.
- A few days ago i tried downloading transactions and this time the error in the OFX file is Invalid ID/Password (Code 15500)
- I am able to log into the account online with the same credentials as in pocketsense.
- I called Chase and had them reset the account. Then (re)authorized the account for Quicken download
- I deleted the account (from setup) in Pocketsense and created it again
- When i try to download transactions, I still get the same error
- Note: I can download the transactions from the webpage; albeit i have to download as QFX, rename as OFX and import into Microsoft Money
Has anyone encountered this problem? Any suggestions on what else i can try?
Hi Sam,
DeleteThe wife's Chase Visa account had a similar problem about 3 weeks ago. However, she could not even log on to the Chase Visa website.
We ended up having to change her password. Actually, we had to change it twice before it would allow her to log off and log back on again.
Once we had her password working with Chase's website, we were able to get Pocketsense to connect again.
You might want to try changing your password even though your issue is slightly different.
-Kevin N.
Kevin -
DeleteThanks! Changed the PWD and it worked!! Would never have thought to try!! Thanks again!!
Hi Sam,
DeleteExcellent! I'm glad to hear that it worked.
Another anomaly in the world of 1's and 0's. :)
Thank you for reporting back with your results.
-Kevin N.
Heads up folks, I'm getting SSL and socket time outs on Chase. Others have complained about it in the past, but this time it may be a known maintenance downtime. According to Quicken web site, Chase is undergoing maintenance 11/9-11/11.
ReplyDelete"Chase has estimated that all maintenance should be completed and connections restored by end of day Sunday 11/11/2018 however this is an estimate and subject to change. "
I've been having trouble with First Tech Federal Credit Union the last couple days, using a very old version (circa 2010) version of the scripts. I changed the script to print out the exception and it complained about SSL. So I updated Python and the scripts to the latest after some other reading I did. Now, it appears their OFX server is trying to send me some HTML loading page. It's literally an HTML page with a title and body that says 'Loading'. If I request an OFX from my online banking account directly, I get the correct data. Any idea what could be happening with this interaction?
ReplyDeleteHi Skippy,
DeleteWhat version of Python did you install?
It is my understanding that the scripts are written to work with Python 2.7.xx.xxxx.
The latest version of 2.7 on the ActiveState website is currently 2.7.14.2717.
-Kevin N.
I have that latest version of ActiveState. I'm guessing this is an issue with my bank. ofxhome.com is reporting OFX and SSL failures for it.
DeleteI've been debugging this for the last couple of days. From a Fiddler trace of Quicken for Windows (which succeeds), it looks like First Tech now requires multiple session cookies to be played back. I tried hacking on both ofx.py and httplib.py in an attempt to set a cookie header that contains the multiple cookies, but then I end up getting a '403 Forbidden'. I haven't figured it out yet. For comparison, Moneydance is choking on the site's certificate, so I can't even tell what it would do.
DeleteThey've been having problems before this week where they would return responses that were missing transactions. I reported it and got a reply yesterday: "We were contacted by you in regards to an error you were receiving with Quicken. We have tested with IT and multiple users and have found that the connection is back up and running."
Because it does indeed work with Quicken, I'm not sure to what extent we'll get support on Pocketsense, our favorite OFX scripts.
Feel free to send a (sanitized) copy of the fiddler trace/log to pocketsense1 /at/ gmail /dot/ com. I'd be curious to see what's happening.
DeleteThey seem to have fixed this on the server side.
DeleteSpoke too soon. Another day, another result. Quicken fails now also with the same HTTP 403. Moneydance hasn't updated their root certificate jar (http://help.infinitekind.com/discussions/online-banking/11390-ssl-issues-with-first-tech-federal-credit-union) so it can't even get an HTTP request through.
Deletereply: 'HTTP/1.1 403 Forbidden\r\n'
So everything seems to point to First Tech's use of Imperva's Incapsula as a WAF (web application firewall). Among other protections, it uses "broken" cookies as a defense against DDos attacks.
Delete"The cookies that Incapsula sends are "broken" on purpose, and they include content whose purpose is to test how the client responds to an irregular cookie - as part of our classification process. While browsers are capable of handling such cookies, most bots aren't, and this is what serves as a first line of defense against them."
I think First Tech is still tweaking their Incapsula configuration, as I get occasional successes via Pocketsense.
Thank you for this info, redmondman! I'm still struggling with them to give me information about what they've done. They keep claiming that they are "OFX compliant" yet "can't tell me about their new protocols" for my "protection". Total runaround. I haven't been able to get pocketsense to work at all since this change. Let us know if you manage to get it working! Having to manually download the OFX files from their online banking and then dealing with dupes is a headache.
DeleteI've got a hacky workaround -- give it a try.
DeleteIn file ofx.py, function doQuery, replace these four lines:
h.putheader('Content-Type', 'application/x-ofx')
h.putheader('Host', self.urlHost)
h.putheader('Content-Length', str(len(query)))
h.putheader('Connection', 'Keep-Alive')
with this block:
if self.urlHost == 'ofx.firsttechfed.com':
h.putheader('Accept', '*/*')
h.putheader('User-Agent', 'InetClntApp/3.0')
h.putheader('Date', datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT'))
h.putheader('Content-Type', 'application/x-ofx')
h.putheader('Host', 'ofx.firsttechfed.com')
h.putheader('Content-Length', str(len(query)))
h.putheader('Connection', 'Keep-Alive')
else:
h.putheader('Content-Type', 'application/x-ofx')
h.putheader('Host', self.urlHost)
h.putheader('Content-Length', str(len(query)))
h.putheader('Connection', 'Keep-Alive')
Oops, that didn't post under the right profile. And I don't know how to format code here, but make sure to indent the h.putheader lines, because ... Python.
DeleteTried the workaround today. Still get 403 Forbidden and "Loading" HTML.
Deleteincap_ses_235_1893025=yeovaa1k8Q1qJi737gNDA9F6IFwAAAAAnEQtGTL
oki0QJboqdsSO1A==; path=/; Domain=.firsttechfed.com
send: 'POST / HTTP/1.1\r\nAccept: */*\r\nUser-Agent: InetClntApp/3.0\r\nDate: Mo
n, 24 Dec 2018 06:21:06 GMT\r\nContent-Type: application/x-ofx\r\nHost: ofx.firs
ttechfed.com\r\nContent-Length: 708\r\nConnection: Keep-Alive\r\ncookie: incap_s
es_235_1893025=yeovaa1k8Q1qJi737gNDA9F6IFwAAAAAnEQtGTLoki0QJboqdsSO1A==; path=/;
Domain=.firsttechfed.com\r\n\r\nOFXHEADER:100\r\nDATA:OFXSGML\r\nVERSION:103\r\
...
reply: 'HTTP/1.1 403 Forbidden\r\n'
I'm confused by the output you posted. Looks like part of the first response and the second request. If all is well, there's no second request.
DeleteIn any case, my best suggestion is to wait 24 hours between attempts. We've seen "stickiness" on the part of the Incapsula WAF in terms of remembering your IP address and its state.
This workaround is still working for me. Feel free to reach me at alan (dot) auerbach (at) gmail.com for more direct troubleshooting.
Thank you redmondman! That fixed the issue I was having with First Tech.
DeleteYay!
DeleteShippy,
ReplyDeleteJust to reiterate, Kevin is trying to let you know not to necessarily install the latest version, which is 3.xx; but instead, just the latest version of 2.7.xx
FYI. Looks like capital one has eliminated their OFX capabilities. http://help.infinitekind.com/discussions/online-banking/11396-download-from-capital-one-360-broken.
ReplyDeleteNot sure of any real problem yet but seems that Yahoo is taking forever to furnish stock and mutual fund quotes. Have not had any issues for about a year now but 7 minutes and only half-way through portfolio of around 50 quotes.
ReplyDeleteVery sad about capital one. I moved all my account to discover bank now. OFX working fine with them.
ReplyDeleteSiteName : DISCOVER BANK
AcctType : BASTMT
fiorg : Discover Bank
fid : 12610
url : https://ofx.discovercard.com
bankid : 031100649
For your account number log into your account and manually download the quicken ofx file. Then look in it at the account number xml node and use that along with your login username and password.
Thanks for posting. I added your info to my blog pages: https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
Deleteand
https://microsoftmoneyoffline.wordpress.com/2017/05/12/discover-has-changed-your-account-id/
Hello,
ReplyDeleteI do a fair amount of shopping at Amazon.com with my Chase Amazon Rewards card. Recently, Amazon changed their billing name to be Amazon.com*(long string that represents the order number). Is there a way to write a routine in the Scrubber file that will strip away the "*(long string}" so that MS Money will alias it correctly when it imports?
-Andrew
Yes. Andrew, if you are looking for help in how you could do that action in your scrubber, I would be glad to help if you have the motivation and ability to make a scrubber that does something, even if trivial.
DeleteOr if you are just offering up a feature suggestion for somebody else to write, you might flesh out how you think that should operate. As a feature suggestion, it could be generalized to allow such translation actions on other accounts.
cl
Thanks cl,
DeleteI'm willing to take a stab at writing the scrubber. Here's my vision in order of generality:
1) Users could specify a "Payee Translation" entry in sites.dat that could be either account specific or across all accounts. It could take "Amazon.com*M12345" and clip it to "Amazon.com" (or even add the remainder to the comment field). Then the MS Money alias feature would work correctly.
2) The code could be present but inactive in Scrubber.py and advanced users could activate the scrubber and manually add a translation table or regex expressions. They would have to re-do this work after every software update.
3) Some working code could be posted and super advanced users would have to re-add it to Scrubber.py after each update.
I'll post my version when I have a chance to try it. If you have the "hard bit" (the code that looks at the payee and the regex expression that clips the Amazon trailing parts) I would appreciate it.
More specifically, I'd like a scrubber subroutine that looks for "<NAME>Amazon.com*M05JI2060</STMTTRN>" style entries and changes it to <NAME>Amazon.com</STMTTRN>.
DeleteHey All,
DeleteI've made a Scrubber routine that removes the trailing portions from the merchant of Amazon.com purchases so aliasing will work correctly in MS Money.
Add this in the Scrubber.py file somewhere:
===============
def _scrubAmazon(ofx):
# 1. Scrub the trailing "*M12345678" from Amazon.com purchases on the Amazon Chase Card
p = re.compile(r'Amazon\.com\*M.{8}',re.IGNORECASE)
if p.search(ofx):
ofx = p.sub('Amazon.com',ofx)
scrubPrint("Scrubber: Amazon.com order numbers removed for aliasing.")
#2. Scrub the trailing "*M12345678" from Amazon Marketplace purchases on the Amazon Chase Card
p = re.compile(r'AMZN Mktp US\*M.{8}',re.IGNORECASE)
if p.search(ofx):
ofx = p.sub('Amazon.com',ofx)
scrubPrint("Scrubber: Amazon Marketplace order numbers removed for aliasing.")
return ofx
===============
Add this in the "scrub" routine, somewhere near the Discover and TRowePrice lines:
===============
if 'CHASE' in siteURL: ofx= _scrubAmazon(ofx)
===============
To make it better, it would look for an option from sites.dat that specifies if you want this done or not. It could also be improved to move that trailing order code to the memo field instead of just deleting it.
Very nice scrubber, Andrew Shafer. Functional and pretty. And you were even able to pretty-print the post.
DeleteIt would be possible to move the extra stuff off to the memo, but hardly important.
cl
Andrew, you could also consolidate the existing Amazon transactions with Tools: Find and Replace. Advanced. Use the text tab, etc.... Specify amazon and look in the Payee field. Then you can change all of the found entries to the existing Payee of your choosing, or you can be selective.
ReplyDeletePlay around with a copy of your file if you want to try that, or keep an extra *.mbf file in case you want to undo something.
cl
I just moved the files over to a new computer and installed Python 2.7.15 but am getting the following error when I try to run Getdata or when I run setup and test my account. Did I mess things up by moving them over from my old computer?
ReplyDelete** An ERROR occurred sending POST request to eftx.bankofamerica.com
Exception type :
Exception val : [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
An online error occurred while testing the new account.
This happens for BofA, Chase and Fidelity accounts, but the others seem ok. I noticed there's some discussion above about similar issues, but I'm not sure I understand what the solution is.
DeleteCMH, I would uninstall your existing python and then install https://www.activestate.com/blog/activepython-2713/
Deletehttps://sites.google.com/site/pocketsense/home/msmoneyfixp1/ofx-automation says you should use the Active Python.
I am running ActiveState 2.7.12, and it works fine for Fidelity Investments.
cl
Thanks, just tried that. Uninstalled regular Python and installed ActivePython latest version but it's still having the same issue for those accounts.
DeleteUpdate - it seems to be working now. Not sure what I did. I was following some of the testing above like opening Certificate Mgr, etc. But ultimately what I think worked was that I logged into fidelity.com instead of netbenefits.com. Not sure how that helped the BofA and Chase accounts work properly, but they do. I also tried logging into netbenefits.com using Internet Explorer instead of Firefox, but that didn't seem to make a difference. Will update if I continue to see issues, but crossing fingers that it's solved going forward.
DeleteCMH, make that the latest 2.7 version.... https://www.activestate.com/products/activepython/downloads/
ReplyDeletecl
I was thinking about how Pocketsense could detect whether a compatible Python version is being used, and to warn in some cases.
ReplyDeleteimport platform
platform.python_version()
Returns the version, which is '2.7.12' for me.
So it could know that it was version 2.7.something rather than 3.something.
Checking that it was not 2.7.15 for now would show it was not ActivePython, but as soon as ActivePython came out with 2.7.15, that would not be useful.
platform.python_implementation() for me returns CPython. I think I am using ActiveState ActivePython.
It does not identify ActivePython or ActiveState however.
cl
I recently reinstalled windows and ran into the same [SSL: CERTIFICATE_VERIFY_FAILED] error mentioned in this thread.
ReplyDeleteUninstalling Python 2.7.15 and installing ActivePython-2.7.14.2717 fixed the problem for me too.
Does anyone else have banks that don't run an OFX server? I've started working on a Python script that lets me automatically download those files.
ReplyDeleteBasic steps:
1) Have ActivePython working.
2) Have Firefox working.
3) Download and add geckodriver.exe (same 32-bit or 64-bit as your Firefox) to your Python27 directory.
Run a script like this (currently for Wealthfront, you'd have to have a separate one for each bank.) It downloads the OFX file to the directory where thy Python script is running. I'll be making another one for Empower Retirement. My ultimate goal will be to get it to be called by PocketSense to download these non-server ofx files, run the scrubber, and combine with the server downloaded files.
import os
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
username = "myemail@myserver.com"
pw = "mysecretpassword"
currentpath = os.path.dirname(os.path.realpath(__file__))
profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2) # custom location
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference('browser.download.dir', currentpath)
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/x-ofx')
driver = webdriver.Firefox(profile)
driver.implicitly_wait(10) # seconds
driver.get("https://www.wealthfront.com/documents")
assert "Login - Wealthfront" in driver.title
elem = driver.find_element_by_name("email")
elem.clear()
elem.send_keys(username)
elem = driver.find_element_by_name("password")
elem.clear()
elem.send_keys(pw)
elem.send_keys(Keys.RETURN)
time.sleep(2)
destination_page_link = driver.find_element_by_xpath("//button[contains(text(),'Get started')]")
destination_page_link.click()
destination_page_link = driver.find_element_by_xpath("//button[@type='submit']")
destination_page_link.click()
time.sleep(10)
driver.close()
If anyone else is interested, I've updated my website with the OFX/QFX scrapers for Wealthfront and Empower Retirement. If you have a bit of web/coding experience it should be pretty easy to create one for own bank if they don't run an OFX server.
Deletehttp://pages.suddenlink.net/shaferandrew/
Andrew Shafer: I had a broker whose website gave a bad OFX file that I had to scrub. My method was a python file, and a batch file that invoked that python file. It fixed the file, and wrote the fix to a file. I could then just open that file, and Money would pick it up automatically.
ReplyDeleteIf you think that would be of interest, I could put the batch file and python file into a zip file, and provide a web link to that zip file. I don't claim that it is particularly interesting. But rather than putting the scrubber into scrubber.py, I made special purpose file. Incidentally, I have a C:\bat\ folder that is in my Windows execution search path, so I can just put batch files in that folder if I like.
So anyway, let me know if it would be of interest.
cl
Anonymous: Is your python file general purpose and able to fix any (most?) OFX files that give the invalid/corrupt message in Money? I have a few different banks that provide OFX downloads but then won't import. Specifically I know I have trouble with Wells Fargo and Capital Bank (First Tennessee) and there are probably others.
ReplyDeleteAny help is greatly appreciated.
Ken: My python file is specific. It is essentially a scrubber for one FI, that I no longer use.
ReplyDeleteThe code for opening a file and writing the fixed OFX could be used for a different FI, but you would want to add the specific fixes for your FI's quirks. I think if you ran my file on a working OFX file, it would not break the file, but if it did not find the particular flaws it knows about it would not really do anything.
I did not try to make it general purpose.
cl (Cal Learner)
How does one figure out what the actual problem is with an ofx file? I've gone over the ones that don't work and can't see anything particular that's causing the issue. I'd post one here but I get a CODE error. I'm guessing the blog software is concerned it looks like malicious code.
ReplyDeleteKen, you could post to a website that lets you upload, such as http://tinyupload.com/
ReplyDeleteThen post the link here.
You usually would want to change the ACCTID with Notepad to 1234567, or some such, before uploading.
cl
Here's the OFX file: http://s000.tinyupload.com/index.php?file_id=94193863965470019019
ReplyDeleteI'd be very interested to understand what you look for when scanning the file. I checked matching open/close delimiters and compared it to a known good OFX file and couldn't see anything. I'd like to be in a position to fix the files myself and help others in the future, if possible.
Thanks!
Ken... I downloaded the file. Two people did it before me. In the text below I use { and } to replace less-than and greater-than symbols.
ReplyDeleteTo make the file easier to analyze, I added a newline before each greater-than AFTER the first. That put everything onto its own line. The file is an OFX 1 file. The analyzer told me to look for an error on line 15. I added 10 to 15, and went to line 25. That was the
Bottom line is that if there is an {FI} record, it needs an {ORG} next.
See OFX 2.1.1 Specification page 53
So here is the modified stuff:
{FI}
{ORG}SnowBank
{FID}FTBTESTID{/FID}
{/FI}
In OFX1 the {/FID} part is optional, but is needed for OFX 2.
cl
Also I see that the link to the "Microsoft Money Analyzer" did not make it somehow. If you Google that, you will find where you can download it. I like to add newlines in front of each greater-than after the first. That lets the line count in the error messages become useful. Otherwise it just reports everything is on line 1.
ReplyDeletecl
One more thing... Even better might be to just delete {FI}...{/FI} -- the whole FI section. I just did that and launched it. Money handled it fine.
ReplyDeletecl
I can't thank you enough. I took out the {FI} {/FI} and it imported perfectly. I also did the line spacing so I could better see what was going on. After that worked I tackled Wells Fargo. Since it was a QFX file, when I saw 2 entries that appeared to be directly tied to Intuit, I deleted them. The file then imported correctly.
ReplyDeleteI downloaded the OFX Analyzer but didn't end up needing it. However, I will keep it for the future. I didn't even know that tool was available.
Next up I'll try and add Capital Bank and Wells to your scrubber. Haven't looked at how the scrubber is structured, but if it's just a straight Python routine I'm sure I can figure it out.
Thanks again. My financial life would be very different and probably impossible in its current form if I couldn't continue to use Money.
http://s000.tinyupload.com/?file_id=09375113273274854716 is zip file. Not a great thing by any means. Just a pair of files I made for myself which could be of interest.. or not.
ReplyDeletecl
I am in the process of moving Pocketsense to a new PC, and I am getting SSL: Certificate Verification Failed, with a code of 661 for Chase and Fidelity. Other banks work fine. On the new PC, I was running Active Python 2.7.13.2715, which was the same I was running on the old PC. I tired upgrading to 2.7.14.2717, and got the same errors.
ReplyDeleteAny ideas?
Per earlier posts here, I tried uninstalling and reinstalling my versions of Python, but the problem stayed with me. What finally worked was my adding a Windows environment variable PYTHONHTTPSVERIFY and setting it to a value of zero. (To do this, go to Control Panel, then System, then Advanced System Settings, then Environment Variables.)
DeleteApparently, this problem is caused by Python changes that began verifying SSL information a few releases back. Why it doesn't work when a new system in installed (and only for some banks) is totally unclear to me, and the workaround indeed does not sound like the greatest of ideas. But I couldn't find any other circumvention that worked for me.
H
Confirming Harold's observation on fresh Miniconda installation on 32-bit Windows 10. Active python no longer supports 32-bit Windows, so have to switch to Miniconda.
ReplyDelete1. Installed and updated Miniconda:
https://conda.io/en/latest/miniconda.html
c:\> Miniconda2-latest-Windows-x86.exe
# updated to the latest:
(base) c:\..> conda update --all
# Test chase account:
c:\...> Setup.py
...
Test CHASE | XXX (Y/N)? y
CHASE : XXX : Getting records since: 20190102
** An ERROR occurred sending POST request to ofx.chase.com
Exception type :
Exception val : [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
An online error occurred while testing the new account.
All other accounts test out fine, only Chase has a problem.
The following short snippet works fine, proving that there is no error with installed certificates:
import requests
url = "https://ofx.chase.com/"
#page = requests.get(url)
page = requests.post(url)
print page.content
Please help.
The script currently makes calls directly via httplib, due to some specific settings that are needed for some banks. The library uses the default ssl connection config, including certificate verification and protocol selection. For better or worse, the "minimum default protocol" can change between python versions, with the intent of making things more secure. I'm wondering if some banks are using older ssl versions for their ofx service? To test this idea, I added a few lines of code to allow connections using versions 2 and 3 of the ssl protocol, which have been disable by default in recent python editions. While not as secure, it's leaps and bounds above no encryption at all.
ReplyDeleteI uploaded an update here, for anyone interested in testing. None of my banks have an issue, so I can only verify that the change doesn't break anything on my end.
At this point, as I somewhat detail below, my PC "has been fixed" and I can't get it to fail. I am hoping that that in itself offers A Meaningful Clue.
DeleteH
Hi Robert,
DeleteThank you for the quick response and sorry about my delay. When trying your modification on both, ActivePython 2.7.1.4 and the latest Miniconda on 32-bit Windows 10, the following error occurs:
Test VANGUARD | 88029325160 (Y/N)? y
VANGUARD : XXXX : Getting records since: 20190106
'module' object has no attribute 'create_default_context'
An online error occurred while testing the new account.
SM
I am sorry, Miniconda did not fail with the above mentioned error. It worked for Vanguard. Instead it failed on Chase as follows:
DeleteTest CHASE | XXX (Y/N)? y
CHASE : XXX : Getting records since: 20190106
** An ERROR occurred sending POST request to ofx.chase.com
Exception type :
Exception val : [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
An online error occurred while testing the new account.
Harold, I suggest that you run the Test with the "urllib" package Python 2 on
ReplyDeletehttps://www.calazan.com/how-to-check-if-your-python-app-supports-tls-12/
The way you run this is to copy that big long code string into a command window command line. To paste into a command window, you may have to set up the command prompt settings, and then right-click. The python help says
"-c cmd : program passed in as string (terminates option list)".
If the pasted command does not run python, you may need to use a fully qualified pathname.
Pocketsense for Fidelity works for me, and it works for you in the old computer. In the test, I get "TLS 1.2" as a result. This test is easy. I am not saying that there is a high probablity that this will point to the problem, but it is easy enough to try.
cl
I tested with the PYTHONHTTPSVERIFY variable set to zero, and I got "TLS 1.2". Then, playing around, I deleted that environment variable, rebooted, and I also got "TLS 1.2". On a hunch, I then tried Pocketsense with the variable not present -- and it worked just FINE for both Fidelity and Chase!
DeleteThis seems somewhat consistent with reports earlier in this section that once you played around and got it to work, it then continued to work. Ultra-weird, imo... I'm going to leave it this way, with the environment variable not set, and see how it goes.
H
Hello, any of you having problem when downloading quotes ? I get an error since a few weeks on all quotes from Google Finance (ex : ** error reading http://finance.google.com/finance?q=GOOG
ReplyDelete** GOOG : invalid quote response. Skipping...)
Any idea ?
Was working great before.
Thank you !
GG
Google Finance hasn't been working since they removed json quote sections from their web interface. I rewrote the Yahoo interface at about the same time (~mid 2018), when they changed their method. Yahoo now supports a much broader quote set than previously, including foreign exchanges.
DeleteThank you for your quick answer !
DeleteI have been a long time user of the OFX scipts. I initially set up my downloads back in 2011 using the info on the Finance Buff site. Everything has worked like clockwork until recently when I was getting errors downloading from Fidelity and T. Rowe Price. I posted my problem on Finance Buff and I was referred to here to switch to pocketsense because of improved scripts. In setup I have successfully added a couple of accounts but I’m having problems in my account set up for Fidelity, T. Rowe Price, Vanguard, and some others but these are the accounts I need the most to work. After I put in my user name and password the script does not connect to the account. I can't see the error messages as the window closes out too fast for me to see them. My user name and password are correct because I can connect to them through the individual accounts. Any help with this is appreciated.
ReplyDeleteSorry I can't edit my previous comment. The version of Python I have is 2.7.2.5 would I need to download a later version or is that not the problem that I might be having?
ReplyDeleteBarry: Definitely need a newer version of Python. I'm running 2.7.15, but 2.7.14 should be good.
DeleteRobert thanks for reply. I realized after posting my first question that I can capture the error message by running in command window. Here are my error messages can you tell me if newer version of python will resolve?
ReplyDeleteAn ERROR occurred sending POST request to ofx.fidelity.com
Exception type : [class ssl.SSLError]
Exception val : [Erno 1] _ssl.c:503: error 1407742E:SSL routines SSL23_GET_SERVER_HELLO:tlsvl alert protocol version
Curious My old scripts with version of Python I have were working fine until recently did something change on Fidelity that would have started this error to happen?
Sorry for multiple questions here. I am not the most computer literate person so do I need to uninstall my current version of Python before downloading the newer version? Or will the one I download now become the newest version that gets run?
ReplyDeleteHi Robert
ReplyDeleteAfter downloading a newer version of python I have gotten through the process so thank you for your time to reply. I have one account problem remaining. For Franklin Templeton I am getting a certificate validation error. I see this error mentioned above in the comments but I do not understand if there was a resolution to this. I saw that one person said that he went down to version 2.7.14 to solve this problem. I had downloaded version 2.7.15 so I uninstalled that and then installed 2.7.14 but still getting the certificate error. If you could kindly reply to let me know if there is something that I can do to be able to download this account I would appreciate it. Thank you for your assistance.
Barry: AFAIK, nobody has posted a site descriptor for Franklin Templeton to go into the sites.dat. You might post yours. That will perhaps be useful for others, plus it would let people see if they note a problem. Change out the pointy brackets for curly brackets, since the forum eats pointy brackets. So it would start with {site} and end with {/site}
Deletecl
I note that you were having problems with Fidelity, but with the newer Python, I presume Fidelity Invesments is coming in fine for you.
So sorry for all these multiple posts. I don't see a way to edit on my previous posts.
ReplyDeleteSo I re-read some comments on certificate verify failure and noticed that the posters said that they downloaded ActivePython 2.7.14 so that is what I did and I still get the certificate verify failure on Franklin Templeton.
My other question is what is the difference between Python and ActivePython? I now have ActivePython so can I keep that or should I uninstall and go back to Python. Thank you.
ActiveState Python is fine. It's a "flavor" of the standard distribution, with additional modules included.
DeleteI haven't personally had the "failed certificate" error, but see Harold's solution above on Jan 28. To make it easier to test, I just now uploaded a simple batch file to run Setup.py with SSLVERIFY disabled. Place this file in same folder as the PocketSense scripts, alongside Setup.py, and run. It will only disable SSLVERIFY for the one session, and re-enable when done. Try to make a connection this way, and if it works, try again using the normal Setup.py routine.
Disabling certificate verification does not affect encryption. The ssl communication will still be fully secure, but the server certificate chain won't be checked. According to other users experience, once you make the first connection w/o verification, it then seems to trust the server (not sure why, but likely caches the certificate). Also, the older scripts and python version you were using before *never* verified certificates, so this is an added level of security. I'd only do this for specific servers that give a failed connection.
Rob thank you for all this help.
ReplyDeleteI ran the Setup No Verify and successfully connected to Franklin Templeton to set up the account, then I continued in Test Mode to a successful completion. Then I ran the regular Setup.py using the Test Option for Franklin and got the certificate verify failure error.
where does this leave me as far as proceeding with this. I can download OFX from the Franklin web site so if you think it is best to let this go then I am good with that.
Harold mentioned rebooting too, but not sure if that mattered. I'm curious about this beyond the occasional issue, since I could pretty easily add an option to skip certificate verify for a specific site/server. I've been reluctant to do so, as I'd rather there were a better way.
DeleteCheck out this post for a recent user who also couldn't connect to Fidelity. His answer, apparently, was to login into his Fidelity account using Internet Explorer (not chrome or firefox), which loaded the certificate to the system. Worth a shot...
I logged into Franklin Templeton with IE. That did not help still get a certificate error. I'll keep an eye out for any updates to this problem but for now I'll download direct from the Franklin web site.
ReplyDeleteHey Robert, is this still an active project? Looks like Canadian MUTF quotes started going flakey last March and stopped working altogether in May. These are the funds I'm trying to get:
ReplyDeleteMUTF_CA:RBF460 s:rbf460
MUTF_CA:AIM1571 s:AIM1571
MUTF_CA:CIF833 s:CIF833
MUTF_CA:RBF1340 s:PHN340
Downloaded the latest version, but no improvement.
You need to convert these from Google Finance to Yahoo Finance. Google Finance apparently shut down their old service within the time frame you observed.
DeleteThere is some information in the 01-Jan-2018 release notes of PocketSense about using Yahoo quotes. There is an EnableYahooFinance entry to add to sites.dat. I don't use quotes in PocketSense, perhaps someone else will be able to help.
I searched for RBF460 to find its name, and searched Yahoo Finance to find their symbol, and cross-referenced it with the RBCGAM site.
RBC Select Balanced Portfolio
Google: MUTF_CA:RBF460
Yahoo: F0CAN05LUD.TO at https://finance.yahoo.com/quote/F0CAN05LUD.TO?p=F0CAN05LUD.TO&.tsrc=fin-srch
Yahoo appears to round to the penny, and not be up to date. It lists a quote for 2/27/2019 of $26.96 as its latest.
Another alternative is to try MSMoneyQuotes from Gaier Software (http://gaiersoftware.com/Money). It retrieves quotes from MSN Money. It is NOT FREE US$9.99, but there is a trial version that you can try to see if that works for you. This program updates your Money file directly, so be sure to make a backup copy first if you want to try it. I have been using this for my quotes since 2014.
MSN: RBF460 at https://www.msn.com/en-us/money/funddetails/fi-F0CAN05LUD?symbol=RBF460&form=PRFIHQ
Hope this points you toward a solution.
That seems to have done it. Thanks Andrew!
DeleteCan I get an explanation for why I get this message in Money when the OFX files are being imported?
ReplyDelete"You cannot perform this operation right now. There is an online operation in progress. Try again later"
Barry: Not sure why, but what I would do is to close money. Set the Windows Firewall to block Microsoft Money from accessing the internet. Then start Money, and try again.
DeleteBlocking this access can speed up Money too.
cl
I just downloaded Python 3.7.2 and the PocketSense files. First, can these files that were configured on 2.7 work in this iteration?
ReplyDeleteAlso, for step 1, creating site file, I've opened the Setup file, but I don't know how to "Select 0" before exiting.
Thanks in advance for any assistance.
TJT asked "I just downloaded Python 3.7.2 and the PocketSense files. First, can these files that were configured on 2.7 work in this iteration?"
ReplyDeleteI very much doubt it, but I could be mistaken.
"Also, for step 1, creating site file, I've opened the Setup file, but I don't know how to "Select 0" before exiting."
If you are trying to exit, you could do Ctrl+C, or probably even press the Enter key.
CL
Thank you. However my real question was "How does one select zero"? (From the instructions). The word select suggests choosing, as opposed to entering, an alternative, such as 0,1,2,3. . Is that correct?
ReplyDeletePlease advise. Thanks again for any feedback or direction.
Hi TJT,
DeleteWhen you "open" the Setup file do you see a file with text like:
# setup.py
# http://sites.google.com/site/pocketsense/
# setup account info
# Intial version: rlc: Feb-2010
#02Mar2010*rlc
# - Corrected bug for entering bank statement type (checking/savings)
# - List account type for all accounts (not just bank types)
# - Cleaned up formatting a bit
# - Added Money Market and Credit Line as valid bank account type
Or do you see text like:
PocketSense OFX Download Python Scripts, Ver: 24-Aug-2018
----------Main Menu-----------
1. Add or Modify Account
2. List Accounts
3. Delete Account (or reset connection)
4. Encrypt account settings
6. Disable Stock/Fund Quotes
7. Test Account
8. About
0. Save & Exit
------------------------------
Selection: [0]
If you see the first, you are opening the file for editing. If you see the second, then you have correctly launched/executed the Python script. You can press "0" or Enter to continue with setup.
TJT. Press the 0 key (zero key) on your keyboard.
ReplyDeleteOften 0 is the default, so in that case, pressing the key is optional.
CL
Yes, I did misunderstand. I was thinking that using Python 3.x was causing a problem where the setup.py was not responding normally.
TJT: My answer was to press the 0[zero] key.
ReplyDeleteMy real question for you is are you being obtuse on purpose? PocketSense does not use a graphical interface. Maybe not your cup of tea.
CL
One observation that I'd like to share is that when setting up new accounts, just because all the accounts for a particular bank show up after entering your username and password, doesn't mean they will actually function if the routing number differs on some of your accounts. If you have alternate routing numbers for some of your accounts for the same bank (CHASE and PNC in my case), set up CHASE ALT1 (as an example) in your sites.dat file with all the same info, but using the alternate routing number, and you should find that your downloads for those accounts will then start functioning.
ReplyDelete-ameridan
PocketSense has been working great. For various reasons, I maintain my own scrubber.py with my own Fidelity scrubbing.
ReplyDeleteI have had some sweep fund (FDRXX) for me, buy and transactions coming in in the OFX. I just modified my scrubber. I will attempt to display just the snippet of code that I used. It is very hard-coded and specific. I thought it possible that it would be useful to somebody, and and the concept could be used for a more general scrubber feature potentially. No action requested by me. Note the formatting I used to post with is experimental, and I am attempting to put a special double-dagger in front of each line in case there is some ambiguity. I expect whitespace to be eaten.
‡ ##########begin removal of FDRXX BUY and SELLs.######################
‡ #remove buys and sells of FDRXX CUSIP 316067107 (hard coded)
‡ # ofx is the string containing the OFX file to modify
‡
‡ (ofx,sub_count)= re.subn( \
‡ r'<(SELLMF|BUYMF)>.*?<UNIQUEID>\s*?316067107.*?</(SELLMF|BUYMF)>' \
‡ ,'',ofx, flags=re.MULTILINE | re.DOTALL)
‡
‡ if sub_count > 0: print sub_count, " FDRXX buys and sells removed"
‡
‡ ##########end removal of FDRXX BUY and SELLs.######################
It's been many years since I needed to deal with this situation (long before Money quit being supported). However, if memory serves, I merged the sweep account position (in your case FDRXX) into Cash. From that point on I no longer got all the buys and sells in Investment Transactions. Instead, my Cash Transactions balance just went up and down. And it didn't require any specific scrubber :)
ReplyDeleteHas anyone else had issues downloading quotes the last couple of days? I can't get Yahoo Finance or Google Finance to work...
ReplyDeleteIndeed - worked fine for me for past 6 months or so and lastly on night of April 15th. Now the script returns "** error reading http://www.google.com/finance?q=CVX CVX : invalid quote response. Skipping..." for each entry, regardless of whether stock or mutual fund quote.
DeleteGoogle quotes haven't worked for quite some time, so I'm surprised it worked recently? They removed JSON data packets from their quote response. I left it in the code as a placeholder, should they decide to re-enable quote support... but I doubt it will happen.
DeleteThat said... use Yahoo. They expanded their quote coverage, and probably have a match for most quotes that were previously only found on Google Finance. Most US symbols are the same, but symbols from other markets are different.
Just tried a fresh install of Pocketsense and tested your original sites.dat file that is created. Ran the Getdata.py command and no quotes were returned ("invalid quote response" was returned on all). I assume that both Yahoo and Google quote sites are enabled but also suspect a change occurred after 4/15/2019 (possibly API or URL handling the requests?). Thanks for all your hard work and continuing to monitor the blogs!
DeleteI'm not sure. I tested just now, and working fine here. Did anything else change on your system. Python version?
DeleteRobert, that's good to know. I had noticed that Google quotes quit working a while ago as well. It's possible that for others it had tried Yahoo finance first and failed, then tried Google finance and failed, and then reported that Google Finance failed (but it was really Yahoo finance that caused the error to appear now). I know I have been using Yahoo finance recently. Nothing else changed recently that I know of (possibly a Windows update?) I'm using Python version 2.7.1 as I have been for years. I'm considering modifying code to read from a csv which I can create from a Google Sheet export pretty easily. Has anyone else ever done something similar?
DeleteBKC, you need to be at a more recent version, like 2.7.14 or 2.7.15
Delete-Dan
NxtTek - Thanks, that did it! I'm now at 2.7.15 and it works as expected.
DeleteThat upgrade (from Python 2.7.6 to 2.7.16) also fixed my issues with errors. Got all quotes from Yahoo! tonight without any problem. Thanks to all for the input.
DeleteThis comment has been removed by the author.
ReplyDeleteRobert,
ReplyDeleteI am wondering if you could try PocketSense with miniconda. All my accounts work except Chase. Chase fails with CERTIFICATE_VERIFY_FAILED as shown below.
The reason I have to use miniconda is that ActivePython stopped supporting 32-bit Windows, while Anaconda still does and is very actively supported.
Thanks!
To install and update Miniconda: https://conda.io/en/latest/miniconda.html
c:\> Miniconda2-latest-Windows-x86.exe
# updated to the latest:
(base) c:\..> conda update --all
(base) C:\misc\ofxpy_pocketsense>python Setup.py
PocketSense OFX Download Python Scripts, Ver: 24-Aug-2018
Password:
----------Main Menu-----------
1. Add or Modify Account
2. List Accounts
3. Delete Account (or reset connection)
4. Change Master Password
5. Remove Password Encryption
6. Disable Stock/Fund Quotes
7. Test Account
8. About
0. Save & Exit
------------------------------
Selection: [0] 7
Site Account Type UserName
----------------------------------------------------------------------
...
Test account #: [0] 1
Test CHASE | xxxx (Y/N)? y
CHASE : xxxx : Getting records since: 20190324
** An ERROR occurred sending POST request to ofx.chase.com
Exception type :
Exception val : [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
An online error occurred while testing the new account.
Miniconda should be fine, so long as it's the latest version. That error has been seen before, but no good answer has ever been found. A fix that has worked for some is to log into Chase using Internet Explorer (not chrome, firefox, etc). That appears to update the certificate in the Windows CA store for the site. The PS scripts do not manage certificates, but rely on the Windows CA.
DeleteRobert, Thank you for the suggestion. It was sufficient only use Microsoft Edge to *connect* to Chase website. The next attempt to test Chase account was successful. There was no need to actually login into Chase using Microsoft Edge.
DeleteI have a few accounts at Chase Bank that work well with Pocketsense, but I'm getting a 2003 error when attempting to download JP Morgan Securities activity for my You Invest account.
ReplyDeleteThese are the settings I'm using, and when running through setup, the account number does show up. Does anyone have this working through Pocketsense that can verify my settings?
SiteName : JP MORGAN
AcctType : INVSTMT
fiorg : JPMS
url : https://ofx.chase.com
fid : 7315
bankid :
brokerid :
appid : QWIN
appver : 2400
mininterval :
ofxVer : 103
Fidelity Investments PocketSense downloads stopped working for me Saturday (error 500). I hoped this was just because something got messed up during some Memorial Day weekend work, and that Tuesday somebody will get things fixed.
ReplyDeleteSearching around, I found the Quicken users have that same problem, and they provided useful info. I was relieved that this was a transient problem.
https://community.quicken.com/discussion/7854260/fidelity-problems
CL
Same problem for me Fidelity not downloading.
DeleteAre you willing to summarize what the Quicken community said? It looks like you have to create a Quicken account to read it and I don't want to do that.
I cannot get in now either. There were maybe 20 posts with 10 or more participants.
ReplyDeleteAn earlier post said that somebody was being called in to address that, but I think that might have been referring to a Quicken post.
A later post said that the problem was that Fidelity had installed some new password software that accidently killed the regular server. The Fidelity Net Benefits server was not affected. Anyway, the expectation was that Fidelity would address this Tuesday.
CL
I just made a successful connection to Fidelity. 6:55PM Eastern USA
DeleteThanks Kevin. I got in too, after seeing your post.
ReplyDeleteFYI, you might find that your latest stock quotes are dated for the 26th rather than Friday the 24th.
CL
Hi CL,
DeleteSeveral years ago I created a separate instance of the PS scripts just for stock quotes. I don't remember exactly why.
I've gotten used to running them separately. One for transactions, one for quotes.
I find that quotes need to be imported before transactions or the quotes won't update. (for me anyway)
I *do* realize that there is a 'SaveTickersFirst' option in sites.dat but, like I said, I've gotten used to running separate instances.
It's an old dog - new tricks thing I guess. :)
Anyway, I'm glad that Fidelity is back in business.
-Kevin N.
As of Monday May 28 7:28a EST my Fidelity account is still not working with Money.
ReplyDeleteHi Joe. Sorry to hear of the problem.
ReplyDeleteWhen you put https://ofx.fidelity.com/ftgw/OFX/clients/download into the address box of IE, expect the title bar to say HTTP 400 bad request. During the weekend outage, the HTTP 500 was the error.
If you look at the OFX file in the XFR folder, the contents may offer a clue. Over the weekend, the file contained Internal Server Error - Read . The server encountered an internal error or misconfiguration and was unable to
complete your request
CL
The OFX file reads in part (HTML cannot be accepted on this site):
ReplyDeleteThe document has moved here.
If I add https://login.fidelity.com/ftgw/Fidelity/RtlCust/Login/Init?AuthRedUrl=https://oltx.fidelity.com/ftgw/fbc/ofsummary/defaultPage to the sites.dat file it still doesn't work.
If I run Setup.py my four accounts all test successfully. If I sent the results to Money Notepad opens with a file named FIDELITYINVESTMENTS20190529072954283117.ofx
Thanks.
I just ran the script again and it seemed to work. However, instead of sending the data to Money, Notepad just opens a file named combined783079.ofx.
ReplyDeleteThanks.