Anyone else having issues downloading from Discover? Just got an error today saying "You don't have permission to access "http:\\ofx.discovercard.com\" on this server. Not sure if this is a temporary glitch, or a change in protocol. Will watch for a bit before acting...
I've been having problems with Discover all this week.... I am getting an HTTP error 403, which is a "forbidden" error, and not being able to connect...
I posted the same comment on my blog too. I haven't had any success by calling their customer service web support. It is difficult to get routed to the right tech support person that can resolve the server issue. I tried claiming I use Quicken, and they say it's their problem. Then I tried saying I use Microsoft Money and they say "we don't support that program". Looking at a QFX file from web downloads doesn't offer much help but it does list the FID as 9625 instead of 7101.
From the Moneydance forum at http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working
Quoting: Posted by robf on Feb 24, 2017 @ 11:03 AM
robf's Avatar I HAVE THE SOLUTION!
The HTTP request User Agent header has changed. In my code (I'm not using Moneydance but my own creation), the Discover server used to accept this header: httpReqst.UserAgent = "httpclient"
Now, if I comment out that header or set httpReqst.UserAgent =null I can download from Discover; if it's left uncommented I get the 403 Forbidden error
The Moneydance software guys (and gals) should update their OFX request with this change for Discover. Be warned, other institutions, such as TD Bank, require the UserAgent header to be set.
Thank you for your attention to this matter. Here are some direct links to the posts from robf on the Moneydance forums regarding his findings with Discover.
Over at ofxhome, robler posted at http://www.ofxhome.com/ofxforum/viewtopic.php?id=47793
Quoting: Here's the full HTTP header list that works for Discover: httpReqst.ContentLength = enc.Length; httpReqst.Method = "POST"; httpReqst.ContentType = "application/x-ofx"; httpReqst.UserAgent = null; httpReqst.ServicePoint.Expect100Continue = false; The first 3 are necessary for OFX requests at any institution. The last 2 are Discover specific, meaning other institutions might require other values.
End Quoting.
I don't know if robf and robler are the same person. According to MSDN, the default Expect100Continue value is true.
I am glad to see I am not the only one having issues - this stared about 7 - 10 days ago for me but my error is slightly different:
** An ERROR occurred sending POST request to ofx.discovercard.com Exception type: Exception Val : {Errno 10061} No connection could be made because the target machine actively refused it.
As others have in this thread, I did reach out to Discover but they only provided assistance for manual download/import.
On a side note, I cannot express how thankful I am to have this site and your help and input in keeping Microsoft Money alive for us Robert! Thank you so much!
Hi, i've absolutely had it with the crap called quicken 2011 & 2014 and determined not to buy 2017. I really don't trust it to add a column of numbers correctly (i've got evidence). Would anyone know a way to export data out of Quicken into Money keeping the categories & data? Got an aweful lot over these 6 years. thanks
You may have the most success by removing most of Money's categories in a blank data file and then setting up new categories in the same fashion as your Quicken has them prior to importing the QIF file created by Quicken. I'm not sure if you should setup the accounts prior to the import or if Money will do that for you.
The expert on the subject is Dan (support at gaiersoftware.com) as he wrote the converter for Money to Quicken back in 2009.
Let me know of your experience and I'll include it in my blog: https://microsoftmoneyoffline.wordpress.com/
It seems BECU has gone through some system changes and now users are unable to connect. http://help.infinitekind.com/discussions/online-banking/5046-boeing-employees-credit-union-aka-becu-has-new-ofx-url has an answer from BECU, but doesn't seem to help.
Tried various combinations and could not get it to work. I've messaged BECU again to try to get the correct url. They've changed this a couple of times in the past and it has been difficult to get the correct path. It appears they are using ofxver 102, so that should not be the issue. KR
Yes... I actually sent a request to @robler, looking for the same info being requested there. I have a few ideas, but it will be more involved to test than I've had time to try... yet.
Got an answer from BECU, I haven't tried any changes yet:
Thank you for contacting us. I’m sorry you’re having trouble connecting to BECU from your external money management program. I believe I can help.
Following the last update to the BECU website last weekend we’ve received reports from other members that connections to their third-party accounting programs were interrupted during the update. To fix this, you should reset the connection. How this is done varies from one program to another, so if you need assistance I recommend you contact the maker of the program you use and ask them to reset the connection to BECU. Make sure they use the following information to download transactions:
If you have additional questions you can reply to this message. You can also contact us by phone during our business hours of Monday Friday 7:00 AM - 7:00 PM PST and Saturday 9:00 AM - 1:00 PM PST at 800-233-2328.
Sincerely,
Jeremy Nelson BECU Technical Support Analyst Phone 800-233-2328
If the above settings don't do the trick... if you use ofxVer: 103 for BECU, then you may need to delete the CLIENTUID line at the bottom of your sites.dat file. That's the only persistent "setting" kept by the scripts, other than the site and account settings.
Just posting this here for reference, so I don't forget. Another (related) post from another site. See https://getsatisfaction.com/quickencommunity/topics/cant-download-discover-transactions-to-mac-quicken-2007-due-to-ol-226-error
"I just got off chat with Discover. They stopped supporting Quicken 2007 as of Jan 2017. They then made some changes in mid-February 2017 (probably Feb 18 as Michael Smith mentioned earlier in this thread) which now keeps Quicken 2007 from working with Discover ... "
That was QWIN ver 1600 which I would have thought stopped working long ago. I think the Pocketsense default version is around 2300, so I don't think that is the issue at all. I've actually tried 2500 as well.
errmsg= "** An ERROR occurred sending POST request to" p = h.endheaders(query) # p = h.request('POST', selector, query, # {"Content-Type": "application/x-ofx", # "Accept": "*/*, application/x-ofx" # } # )
errmsg= "** An ERROR occurred retrieving POST response from" #allow up to 30 secs for the server response (it has to assemble the statement) h.sock.settimeout(30) response = h.getresponse().read()
f = file(name,"w") f.write(response) f.close()
Running this version of ofx.py in Setup.py option 7, test mode, only for my Discover accounts yields (private info rewdacted)
DISCOVER BANK : ********** : Getting records since: 20170131 send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 680\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\nOFXHEADER:100\r\nDATA:OFXSGML\r\nVERSION:102\r\nSECURITY:NONE\r\nENCODING:USASCII\r\nCHARSET:1252\r\nCOMPRESSION:NONE\r\nOLDFILEUID:NONE\r\nNEWFILEUID:************************************\r\n\r\n\n\n\n20170303103349\n**PRIVATE**\n**PRIVATE**\nENG\n\nDiscover Bank\n12610\n\nQWIN\n2500\n\n\n\n\n\n************************************\n4\n\n\n031100649\n**********\nCHECKING\n\n\n20170131\nY\n\n\n\n\n' reply: 'HTTP/1.1 403 Forbidden\r\n' header: Server: AkamaiGHost header: Mime-Version: 1.0 header: Content-Type: text/html header: Content-Length: 269 header: Expires: Fri, 03 Mar 2017 15:33:50 GMT header: Date: Fri, 03 Mar 2017 15:33:50 GMT header: Connection: close header: Set-Cookie: ... Invalid OFX statement. ** Review .\xfr\DISCOVERBANK20170303103349331974.ofx for possible clues... An online error occurred while testing the new account.
Note, NO User-Agent header, no Expect header. Still get 403 response. These are the same headers sent by the original code, slightly different order.
Here is hard-coding the query to match the one posted by robler on infinitekind. I hard-coded my user name, password, and account number in the query (redacted in output) as query = '...' just before the try:.
DISCOVER : **************** : Getting records since: 20170131 send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 720\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\n\r\n\r\n\r\n\r\n20170228232807\r\n**PRIVATE**\r\n**PRIVATE**\r\nENG\r\n\r\nDiscover Financial Services\r\n7101\r\n\r\nQWIN\r\n2500\r\n\r\n\r\n\r\n\r\n0\r\n\r\n\r\n**PRIVATE**\r\n\r\n\r\n20130101000000\r\n20170228232807\r\nY\r\n\r\n\r\n\r\n\r\n' reply: 'HTTP/1.1 403 Forbidden\r\n' header: Server: AkamaiGHost header: Mime-Version: 1.0 header: Content-Type: text/html header: Content-Length: 269 header: Expires: Fri, 03 Mar 2017 16:04:13 GMT header: Date: Fri, 03 Mar 2017 16:04:13 GMT header: Connection: close header: Set-Cookie: ... Invalid OFX statement. ** Review .\xfr\DISCOVER20170303110412992001.ofx for possible clues... An online error occurred while testing the new account.
Sending his working OFX query still yields a 403.
Until robler can post his/her header's, I don't think we will get anyplace.
Finally had a little time yesterday to actually focus on the Discover issue. I cobbled together a small .net module to test connections using their http library. I was able to get a successful connection and ofx response, but I tested using OFX 2.x (xml), so still need to test with sgml. Initial (quick) comparisons between the python httplib debug output vs .net output didn't reveal anything obvious, so more to do...
I was hitting the same thing, but thanks to robler's notes, I was able to get it working again as well. The server is very finicky about the order of the HTTP headers -- this sequence works for me: POST / HTTP/1.1 Content-Type: application/x-ofx Host: ofx.discovercard.com Content-Length: [the length of the request] Connection: Keep-Alive With those headers, the XML request robler posted above works for me. Hope this helps.
End Quote
I was able to confirm that this order works in pocketsense for Discover Bank for both bank accounts and credit cards. This is still sending the normal ofxsgml headers. I made this change in ofx.py in function doQuery().
***** CAUTION: THIS HAS NOT BEEN FULLY TESTED WITH OTHER BANKS ***** Let Robert supply a tested version that he confirms works with other banks also. ********************************************************************* try: errmsg= "** An ERROR occurred attempting HTTPS connection to" h = httplib.HTTPSConnection(host, timeout=5) # Uncomment next line to turn on debugging info # h.set_debuglevel(1) h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1) h.putheader('Content-Type', 'application/x-ofx') h.putheader('Host', host) h.putheader('Content-Length', str(len(query))) h.putheader('Connection', 'Keep-Alive')
errmsg= "** An ERROR occurred sending POST request to" p = h.endheaders(query) # original code # p = h.request('POST', selector, query, # {"Content-Type": "application/x-ofx", # "Accept": "*/*, application/x-ofx" # } # )
errmsg= "** An ERROR occurred retrieving POST response from" #allow up to 30 secs for the server response (it has to assemble the statement) h.sock.settimeout(30) response = h.getresponse().read()
Excellent. I actually went down a different path when I started focusing on the issue. Discover appeared to be fully supporting OFX 2.x for direct-connect, so I assumed it would eventually become the "standard". That may not happen, but I started adding support for OFX 2.x messages (xml), and got that working. The catch is that Money doesn't support OFX 2.x, *but* the difference between sgml and xml is minor. Replacing the message header with a 1.x version gets past the Money import "format check", and the extra trailing field closures are accepted (although optional) in sgml. Seems to work fine.
I had assumed that I'd either be adding a compiled module for the http connection, or writing a sockets interface in the scripts (for full control). If the manual httplib request you've used works reliably, then that's even better!
Ditto Andrew - I made 2 setup.py/ofx.py combos for Discover/non-Discover accounts and am back in business (I prefer downloading per-account rather than in a batch). MUCH appreciated for both of y'all's work!
The key was robler's comment about the required options for the .NET HttpWebRequest class -- once I got that working I was able to log the HTTP request. It's so sensitive that I wasn't even able to get the http library I was previously using to work and ended up writing a custom client just for this server.
End quoting
pgs was able to see the debug the http headers in .net and found the order that worked. This despite the standard stating the header order should not matter for unique headers!
I took a look at the generated ofx responses. Discover Card is still broken the same like {FITID}FITID20170216408.850 after scrubbing. Discover Bank is still broken the same like {FITID}SDF7957870 after scrubbing.
**** CAUTION **** I strongly urge that if you don't need to download Discover transactions today, you should wait until Robert has an official release. **** CAUTION ****
NOTE: The previous post was deleted because a debugging line that might change the behavior was left inadvertently.
This should be a safer change for the doQuery() function in ofx.py. It explicitly treats Discover stuff separately. Note that if you want to try it, you need to correct the indentation which is stripped by the board.
def doQuery(self,query,name): # urllib doesn't honor user Content-type, use urllib2 url = FieldVal(self.site,"url")
isDiscover = 0 if 'DISCOVERCARD' in url.upper(): isDiscover = 1
try: errmsg= "** An ERROR occurred attempting HTTPS connection to" h = httplib.HTTPSConnection(host, timeout=5) # h.set_debuglevel(1)
errmsg= "** An ERROR occurred sending POST request to" if isDiscover: h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1) h.putheader('Content-Type', 'application/x-ofx') h.putheader('Host', host) h.putheader('Content-Length', str(len(query))) # h.putheader('Accept', '*/*, application/x-ofx') # h.putheader('Accept-Encoding', 'identity') h.putheader('Connection', 'Keep-Alive') p = h.endheaders(query) else: p = h.request('POST', selector, query, {"Content-Type": "application/x-ofx", "Accept": "*/*, application/x-ofx" } )
errmsg= "** An ERROR occurred retrieving POST response from" #allow up to 30 secs for the server response (it has to assemble the statement) h.sock.settimeout(30) response = h.getresponse().read()
f = file(name,"w") f.write(response) f.close() except Exception as inst: self.status = False print errmsg, host print " Exception type:", type(inst) print " Exception Val :", inst if response: print " HTTPS ResponseCode :", response.status print " HTTPS ResponseReason:", response.reason
I was late getting home, but had time to test the updates done thus far. In a nutshell, the new version builds the ofx header using the rules posted by Andrew above, and worked with all of the sites that I personally use. Additionally, I added support for OFX 2.x, and successfully tested with Discover. Actually, I tested Discover with ofxVer=102, 103, and 211, and all worked fine w/ Money. Discover bank downloads appear to be offline at the moment, so I couldn't test import to Money, but connections worked fine earlier when I first wrapped up the script edits.
I've posted a beta version @ https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip . If anyone has time to test using their account settings, that would be helpful. I'd recommend stepping through each individually using the Setup module, rather than all at once with Getdata.
All of mine worked flawlessly, including Discover Card and Bank:
TDAMERITRADE USAA FEDERAL SAVINGS BANK (Checking) USAA FEDERAL SAVINGS BANK (Savings) USAA CC (Amex) E-TRADE INVESTMENTS OPTIONSXPRESS DISCOVERBANK DISCOVERCARD CHASE CC CITIBANK COSTCO
I have been using Pocket Sense ever since Money abandoned us. If I am techie at all it would be level 1, but I somehow managed to set it up reading and rereading instruction. I have not a clue what most of this discussion means, but I knew Discover screwed it up and wouldn't help, however I took a chance and downloaded the beta file and managed to set it up again (a long time since I set it up first time). It works with my Citi and my Discover. I don't know why, but thank you, thank you very much is all I can say. Rette
Hi Robert ... Just tested your latest zip and it works fine on all my Institutions just like the current version does.
One new wrinkle just showed up today.....Citibank Fails with INVALID OFX both current version AND new beta version. I am able to download with win phone version of my software, but not pocketsense. I suspect something Citi changed over the weekend as everything worked before Monday morning.
Looks like banks are making changes to keep us occupied.
Any chance you can fold in the siteClientUID changes previously proposed as well? I've been running Pocketsense with those changes manually added just in case Schwab ever switches to ofx 1.03 and I have to keep manually adding the changes in for each revision. It's a nice revision for those that have multiple accounts at one ofx 1.03 institution.
thanx, ameridan
-------------------------------------------------------------------------------- --- OfxPy/orig/ofx.py Wed Feb 11 10:07:36 2015 +++ OfxPy/ofx.py Thu Feb 11 23:37:36 2016 @@ -86,7 +86,14 @@ clientuid="" if "103" in self.ofxver: #include clientuid field only if version=103, otherwise the server may reject the request - clientuid = OfxField("CLIENTUID",userdat.clientuid) + + # if a site-level clientUID is defined, use it, otherwise default to global clientUID + clientuidval = userdat.clientuid + if FieldVal(site,"siteClientUID") <> '': + clientuidval = FieldVal(site,"siteClientUID") + + clientuid = OfxField("CLIENTUID",clientuidval) + print "using ClientUID of: ", clientuidval
Hi Dan: Can you refresh my memory on this one? I recall adding support for multiple accounts at one institution, where they all have the same username. Is this related, or different?
I think Dan may be referring to the issue that came up at Chase that forces us to run two instances of Pocketsense because of some GUID (or something like that) field?
This is that issue, but I don't recall you actually incorporating it. Here's the link to the author's site to jar your memory: http://pastebin.com/9UJ333RZ
Do you think ofx 2.0 will be next for some of these financial institutions, rather than 1.03? If so, I guess we could hold off till it actually happens, but I thought the author (forget who) had come up with a nice solution.
------------------------------------------ By the way I just tested your Beta at Schwab, and Money didn't like it "The file you attempted to import appears to be invalid..."
Here's my output file header: OFXHEADER:100
DATA:OFXSGML
VERSION:2500
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:dd1a870f-23a5-47dc-bae7-30ec160c1139
vs. my file header from Friday: OFXHEADER:100 DATA:OFXSGML VERSION:102 SECURITY:NONE ENCODING:USASCII CHARSET:1252 COMPRESSION:NONE OLDFILEUID:NONE NEWFILEUID:NONE
I set the default version to 2500... to see if it "broke" anything. Guess it did :) I wondered if servers did a "greater than or equal to", or looked for specific values. Try Schwaab w/ a lower value, and let me know.
The beta worked for Schwab; it's Money that didn't like the file and that was the only difference I could find, But I substituted the entire header in today's ofx file and Money still rejected it, so the version wasn't the problem.
The non-beta download works for Schwab? If so, will need to compare the message body, since I don't think I changed anything in the request, except when ofxVer=2xx.
I think I'm confused. I set default appVer=2400, but ofxVer should default to 102 if not specified in sites.dat. I missed that above (appVer vs ofxVer).
Does Schwab work w/ the beta when ofxVer=102 and appVer=2500?
Excellent catch Robert!! I apologize ( I was confused too seeing VERSION:2500). I was experimenting in December and mistakenly entered 2500 in my sites.dat file for Schwab ofxVer, rather than appVer. Until the beta, it was ignored I guess. All is well now with Schwab and your beta. YEAHHH
I found the discussion re: possibly adding site-specific ClientUID values, and remember why I didn't do anything. It's one of those "easy to do but messy" hacks. If I'm understanding, the underlying issue is that each username needed it's own UUID, or more likely, the bank is associating a UUID with a username, and doesn't allow it to be reused. I tossed out the idea of adding a site-specific ClientUID field, so a user can have multiple copies of a site, and use one for each unique username. It was an easy fix from a code perspective, but messy to use and not well understood, so I left it alone. What I hoped to learn (but didn't) was whether a site required a username to have the same UUID across all accounts, or if it was ok to assign unique UUIDs per account setting. That would be an automatic (elegant) solution.
I'm getting the following error when testing it with Setup.py:
** An ERROR occurred sending POST request to ofx.discovercard.com Exception type: Exception Val : endheaders() takes exactly 1 argument (2 given) An online error occurred while testing the new account.
Kevin, did you overwrite control2.py, ofx.py and rlib.py? I got that message thinking that all the changes were in ofx.py but apparently the other 2 files are needed too.
I replicated your steps, and it worked for me. Before tracing out differences, what version of Python are you running? I tested using Python 2.7.12 and 2.7.13.
No, I don't actually have a way to test it; I just added it to my previous version thinking that soon ofx 1.03 would be implemented by more financial houses. Like Robert implied, it is easy for someone that has a need for it to add, rather than having separate Pocketsense implementations for each additional account.
I kinda' wish Robert would add the code, as it doesn't get in the way at all if not needed. Unless he has a more "elegant" solution. ;)
Here's my blog article on implementing it for others to reference. https://microsoftmoneyoffline.wordpress.com/2016/04/26/resolving-errors-for-those-having-multiple-accounts-at-chase-and-other-ofx-103-banks/
I don't mind adding it, and it's easy to do. I originally had thoughts of automating, creating unique uuid values for each client/username/account combo, but wasn't sure if the uuid should be the same for multiple accounts under the same username? The idea was to create a (hashed) lookup table of unique pairs, adding new ones as they come along. It wouldn't be human "readable", but there wouldn't be any extra config.
I downloaded and setup the beta version but am still receiving the same error. I did install Python 2.7.13 (from 2.7.5) and still no luck:
** An ERROR occurred sending POST request to ofx.discovercard.com Exception type: (less than sign)class 'socket.error'(greater than sign) Exception Val : {Errno 10061} No connection could be made because the target machine actively refused it.
Hi Robert - was out of town the last couple of days so I apologize for the delay in my response. To answer your question, yes, it only happens with Discover - I have it setup for AMEX and it works fine.
Regarding the use use "user specific" ClientUID values, before adding the site->clientUID option, I'd like to understand what we're trying to fix. I know that some folks are using a custom UID value for each username (his and hers, if you will), but which of the following requirements is true:
a) a unique UID value is needed for each site+username combo b) a unique UID value is needed for each site+account combo, or c) a unique UID value is needed for each site+username+account combo
In my case with Chase Credit Cards (his and her accounts) Obviously, the site is the same, the account numbers are different and thus the usernames and passwords are different.
The (hers) account uses a unique siteClientUID. The (his) account uses the global ClientUID Both use ofxVer 103
I retained the siteClientUID and the global ClientUID from my prior version of the scripts. In the past, changing the ClientUID triggered a prompt from Chase to verify my ID at their website. I'm thinking that a new or different ClientUID appears as a new computer / connection and they need to add that connection as a trusted connection.
Since Fidelity (and maybe others) allow the same account number for mulitple usernames, I'm going to say that keying to the site and username is the safest bet. My preference at this point will be to simply add unique clientUID values based on site+username, and store them in a lookup table/file. The site and username would be hashed, for security. This approach would be fully automatic, with nothing special by the user. If it turns out we need site+user+account lookup, then it's easy to add. Sound ok?
Sounds good to me Robert. Sounds like Kevin may have to make another verification call for their Chase accounts (as an example) unless the table allows overriding the automatically generated clientUID values with their existing ones.
It seems that when Chase detects a new computer / connection it triggers a notification email that requires the user to log onto the Chase website. Then, through a secure message, the user clicks on a 'Verify your identity' link.
My only concern is that they [Chase] may limit the number of trusted connections to any given account. I'll look further into that on their site.
I sent a message to Chase regarding the number of trusted connections / computers that can be assigned to any given account.
I won't bore you with the somewhat boilerplate response. Essentially, all that is needed is to verify my identity with each new computer / connection / ClientUID.
If I understand you correctly, your plans are to create a separate table for ClientUIDs. Can that table remain intact with subsequent upgrades to the scripts? If so, then that would be ideal.
Robert, Looking at the Beta reminded me of two scrubber issues.
The first had to do with Discover and the strange $0.00 "second line in statement" transactions that it downloads. Did you ever come up with a scrubber for that? I remember reading about it years ago.
Second we had discovered that some 401k downloads (Net Benefits, for instance) would put the wrong sign in front of reinvestment transactions, and Money would ignore the transaction. I wrote a quick scrubber for it a while back. Are you interested in incorporating it to the main trunk?
1. I had decided not to ignore the $0.00 transactions, since they usually contain additional transaction info. I'm not opposed to adding a filter option, but that was how I'd left it before. 2. Pretty sure the "wrong sign" issue has been addressed in scrubber.py (?).
Hmm. Most of my $0.00 transactions usually have some random, 11-character, meaningless code. Not a big deal either way. I could see how sometimes the $0.00 transaction may be useful
The version of scrubber.py I have shows a change by you from 2013 that addresses INVBUY or INVSELL transactions. From the comments I made in the file year+ ago, it looks like I copied your_scrubINVsign() functions to create a new function that fixes REINVEST transactions.
And once again, thanks for all the work you've done with this. It's amazing we are still able to use MS$ eight years after it went off the market.
On Discover: yes, some transactions have extra info, and many do not. Contrary to what NxtTek say, these are not pre-authorizations, as they can show up days after the related transaction. They are some "extra" info that Discover includes in their statements, like cash back, but for some reason when you download they just show up as a meaningless code.
On Fidelity: Yes, I have an equivalent fix for REINVEST transactions to the one you have for INVBUY and INVSELL.
"$0.00 transactions usually have some random meaningless code." These are pre-authorization transactions, meaning a real transaction will probably be occurring the next day. Since I download on a daily basis, their existence can be informative for fraud detection if you haven't ordered anything.
On Discover, I've found that it sends a $0.00 transaction only when it is a transaction that is eligible for their increased cashback offer for that quarter. Citibank occasionally sends a $0.00 transaction, but it is usually for something like a "membership charge" which my account doesn't actually get charged. None of my other banks sends $0.00 charges.
Hi, Robert... if you're considering including minor improvements, the change that I've made locally that I find useful is retrying a bad HTTP return code 5 times. For each time, I put out a line telling what the return code I got was. (Even with this 403 thing that is coming back now from Discover and Citibank, it retried five times before giving up.) I can't tell you why, but various banks occasionally come back with a bad HTTP return code, and just retrying it makes it work fine. Chase does this about once a month, and other banks seem to do it, too, from time to time.
I'd be ok with specific retry reasons, but there are legitimate causes where I probably wouldn't want it retrying over again (e.g., bad url). Do you have some specific urllib exceptions that you see?
I've gotten all kinds of responses over time, including some that made no logical sense. So I check response.status... if it's anything but 200, I go to sleep for a second, close the connection, and loop back to re-establish the connection. And I let the loop run five times before giving up. It has made the process more "robust" for me, for lack of a better word.
I've posted a new beta version @ https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip .
A summary of changes: - Changed clientUID to be specific to the user and bank url. It's "automatic", with no special user config. Note that the first time you connect, a new clientUID will be created, but then reused indefinitely, unless reset by the user via Setup (see below). - Modified Setup.py to allow "resetting" an account connection (clears clientUID for user@site). Deleting an account entry also resets the connection. - Added skipZeroTransactions: Yes/No option to sites.dat. If Yes, $0.00 transactions are removed from statements. Default=No - Added REINVEST transaction types to _scrubINVsign()
Feel like I'm forgetting something, but I think that's the major edits for now, plus a few bug fixes. Feedback appreciated.
I just wanted to confirm with you that the 03-12-2017 beta version of the scripts work as expected with: Chase Visa Discovercard Fidelity Regions Bank
The 'Test' using Setup.py for Chase *did* initially throw an error until I verified my identity @ Chase. The $0.00 amount scrubber seems to also work as expected with Discovercard. I tried the new 'Delete or Reset' option in Setup, up to the point of where I could (C) Cancel.
I'd say that you've hit another home run. Well done!
I can't thank you enough for your work on the scripts.
Worked with all but my citi accounts but I think citi discontinued their OFX recently as I couldn't get it lately with old version. Worked with: Chase, Discover, CapOne360, Vanguard, AmexStarwood A note on Chase:: I had to re-authorize it after first run...get email, go to secure message and follow instructions.
If anyone can confirm...is Citi OFX dead..might be time to kill a credit card or two.
My bad. Citi is working fine. Not sure what was going on but currently on the beta it is working great. Thank you for all your work...now if only MS would open source MSmoney.
Worked fine for all my accounts. Since you're looking for feedback thinking you forgot something, the only very minor things I noticed: 1) last line of Revisions in sites.dat: # -Added skipZeroTransactions
2) your code still addends generic clientUID to sites.dat which may very well be intentional
Works fine here, Robert. Including AT&T (run by Citicard and stopped working with last full release version a few weeks ago, Chase (required an online verification), Fidelity, and others for which I have not seen reports of issues here. Thank you for continuing to support this invaluable add-in.
First, thank you so much for continuing to support PocketSense! It's awesome being able to still use Microsoft Money after all these years.
I just upgraded to the new beta after my Citi credit card transactions stopped downloading with the older versions of PocketSense. That fixed the problem and everything is working great.
However, I did find one small bug in the beta. I tried removing an account and I deleted it from the sites.dat file before removing it from PocketSense using setup.py. When I went to remove it from PocketSense, it would crash and close without removing it. To get it to remove successfully I had to put the site back in my sites.dat file and remove it in PocketSense first, before deleting it from the sites.data file, again.
Thanks for the feedback guys. A couple of notes: - I left the clientUID in sites.dat in case someone reverted to an earlier version... and it isn't used by the new version, so no harm, no foul. - I just now modified Setup to only delete the connection key if the site still exists in sites.dat. That corrects the crash "bug", but... if a person deletes the site before clearing the account, there will be dangling connection keys in the connections file (connect.key), since it can't delete the key without knowing the url. Not recommended ;)
I need to upgrade for Citibank, not sure I want to. Does this mean We can no longer control what the ClientUID is and have to accept whatever is generated? and how do we get it's value. I need the value for other software so I don't have to go through the new clientUID routine each time.
I'll add an option in Setup when listing accounts to include the clientUID in the list. Once created, it won't change. They are stored in connect.key. The username and url are "hashed", but the clientUID is not.
The beta originally seemed to work fine for me. I had to re-register both Chase and Capital One, but it was easy enough, and the new code allowed me to get rid of my two PocketSense directories. Citibank and Discover both initially seemed to work well. However, oddly enough, when I ran the full download, my Discover Bank entries, which I had entered manually earlier in the month, did not make me go through the normal Money sequence of matching downloaded transactions to my manually entered transactions... it also showed in the account register as unmatched. However, oddly, if I ran Setup and Test Account, it worked fine, even if I did more than one in the same run of Setup. (I left myself two more Discover Bank entries to test a lter version of Pocketsense.)
I have no idea how to code in Python nor do I understand OFX "stuff". However, I retrofit my old changes to your beta ofx.py DoQuery routine to retry five times in case of an invalid HTTP response code. I noticed that the old code used to have a statement that said "if h: h.close()" and the new code doesn't have that... I have no idea what it was supposed to do, nor if it was simply unnecessary, but in light of the post above, I thought I'd just point that out.
Harold, my Discover transactions are all matching up fine. Perhaps you downloaded some from the web? Those didn't match up for me after the beta trial, I think because the scrubber routine makes some changes to the FID (or datestamp?). I had to manually match those that I had downloaded from the web, but all is well again now.
I entered them manually. And it seems really weird to me that it works and matches up when I am running from Setup, but it seems to ignore the downloaded transactions when I run Getdata. It certainly seems like it might be the scrubber stuff, but why should that make a difference?
It is odd, since the process is identical... *unless* you enabled CombineOFX, and are packaging all statements into one. If so, I'm guessing that Money is somehow interpreting it differently. I use Discover also, but never entered anything manually during the "outage", and I do use CombineOFX. No issues since resuming downloads.
So keep in mind that I don't understand this stuff except generally. I tried adding the h.close statement, but that didn't eliminate the problem. Then I tried deleting the transaction, and adding it back to Money manually, and I got the same behavior... it didn't work with Getdata but it did work with Setup.
So then I looked at the downloaded files, and lo and behold the one from Getdata did not have the transaction and the one from Setup did.
The obvious clue that I can think of is that the DTSTART and DTEND values for the file that came as a result of Setup were 2/11/2017 and 3/1/2017, while the values for the file that came as a result of Getdata were 2/28/2017 and 3/02/2017. The problem transaction was a end-of-month transaction with a posting date of 2/28/2017.
I have no idea how or why these dates were set.
I am NOT running CombineOFX. I am running with defaultinterval defaulting to 14, with no overrides. The transactions are for Discover Bank.
I think I "got it" almost as soon as I hit enter on my prior post. Today is March 15th, so with defaultinterval=14, I should NOT get the transaction from 2/28. I suspect that Setup uses a different logic to specify the start date for transactions and that's why I got the transaction when using the Setup test function.
I believe you do. I misunderstood the original issue, thinking you were getting *duplicate* downloads w/ Getdata, that weren't matching with originals. The download interval for testing in Setup is 31 days, w/ the hope of catching at least one transaction over the previous month.
Some minor bug fixes I noticed, and added the option to show clientUID (connection key) when listing accounts in Setup. I think it's about ready to put out as "final", but that doesn't mean other ideas won't be considered for adding (e.g., retries, etc.).
errmsg= "** An ERROR occurred retrieving POST response from" #allow up to 30 secs for the server response (if it takes longer, something's wrong) h.sock.settimeout(30) response = h.getresponse() if response.status == 200: # Good response code RetryTime = RETRY_TIMEOUT + 1 # Terminate loop respDat = response.read()
#if this is a OFX 2.x response, replace the header w/ OFX 1.x if self.ofxver[0] == '2': respDat = re.sub(r'<\?.*\?>', '', respDat) #remove xml header lines like respDat = OfxSGMLHeader() + respDat.lstrip()
f = file(name,"w") f.write(respDat) f.close() DOWNLOAD_SUCCESS = 1 # Set flag else: RetryTime += RETRY_INTERVAL print " +Error response from server:", response.status, " Retry time:", RetryTime time.sleep(RETRY_INTERVAL) h.close()
except Exception as e: RetryTime = RETRY_TIMEOUT + 1 # Terminate loop self.status = False print errmsg, self.urlHost print " Exception type :", type(e) print " Exception val :", e
OK... OK... so did I know that this posting board would get rid of all indentation, which is relevant in Python? I don't think there is a way for me to delete a message, or I would delete my sample code.
Robert, Many thanks for keeping this invaluable add-in functional. I ran into a problem with my AT&T card (a CitiCard brand) which was fixed by this latest beta. My only problem is the few institutions that don't have OFX servers.
Harold: I took a few minutes to look at adding a retry option, but I can't test, and I'm not sure what conditions you've had where it makes it to the response check (except an actual response). The only issue I've seen (and can reproduce) are timeouts, which throw an exception.
I hear you, Robert. Even since putting in the beta, I've been having good days lately, with no errors. If you want to test, though, you can just go back to the version of Pocketsense before the beta and try Discover. Of course, the 403 error will turn out to be non-recoverable.
OTOH, it's really not a big deal. When I get your code, it typically takes me just a few minutes to integrate my changes.
Thanks a lot for listening, and for keeping this great software going!
Just tried the latest beta. Observations: - All accounts connected fine - I did have to re-register with Chase, which interestingly it was not required with the last beta. - The Discover $0.00 scrubber works great. Since it is a global setting, rather than an account setting, I'll keep an eye out to make sure there are no unexpected impacts to other accounts. - Cannot test the REINVEST sign issue until there are new REINVEST transactions ready to come down. - A small handful of transactions in two different accounts are not getting matched. Even after I deleted the old downloaded copy and accepted the new downloaded copy. I will have to wait a day or two to see how widespread the problem is.
Weird Chase issue this (Sunday 3/19) morning.... with next to last beta... CHASECREDIT returned {CODE}2000{SEVERITY}ERROR{MESSAGE}ofx.validationException] sporadically (I replaced angle brackets with braces to allow it to post). It would come back this way... then work fine... then come back this way.... unpredictable. It worked fine last night and last week...
After running the scripts, I ran Moneydance, which also uses OFX direct-connect protocols and it threw the same error that you reported.
The fact that it's happening sporadically across more than 1 application would make it appear to be a Chase issue and not an issue with the scripts themselves.
It seems to be working now, some hours later. It's reasonable to assume that it was Chase, although it might be useful to know if Pocketsense was actually sending exactly the same thing and getting different results.
Is there any easy way to capture the data Pocketsense sends to the financial institution?
Something has changed at Chase since Sunday. Pocketsense does not appear to be affected. I have seen errors in other software (The error shows up as NOT IMPLIMENTED.), but this software continues to run fine. I have been using the combined ofx file and have not noticed anything out of the ordinary.
I used to download my Discover statements through Pocketsense. Then it stopped working in January as everyone here has noticed. Then last week Discover changed my CC account number. When I now try to download my transactions using the latest Pocketsense beta, the OFX file has an error that I'm not authorized to access the account and I need to register my account "through the online account center". I've already confirmed that I'm using the correct new CC, username, and password in Pocketsense.
Discover helpdesk was useless because I'm not using Quicken. I did manage to find a registration page by digging into the discover.com front page, but when I register my card there, it tells me I'm already registered and offers to change my password. Does anyone know where I can register my new Discover CC for OFX access?
Any luck with getting TD Bank to work with these changes? It requires the useragent to be set. Not sure if it worked with previous versions because I just got a TD Bank account.
Hmmm. I tested a the sites I use with and w/o a user-agent defined. All but Discover work with or without. The decision, then, is whether this should be a parameter that defaults to off or on. Since Discover is probably one of the most widely used, and most servers don't seem to care, I'm leaning towards a site option to enable the user-agent field, but default to none.
Scratch that. I'm just going to implement a retry w/ the user-agent enabled, if it fails without. It may be that a site requires a specific user-agent name. If so, I'll just add it as a site option.
Any help you can give with getting Merrill Lynch to work is appreciated. Pardon my ignorance but what exactly is the user-agent field? Is it something specific and unique to each financial institution? If so, where might they be listed?
I didn't know either, but apparently it is part of the request header and identifies the application requesting the information from the server, typically a browser, and helps the hosting server determine the best way to provide the requested information. Robert may have to use a dummy browser identifier??, since a Python script is actually making the request using the appID=QWIN, identifying the requesting App as Quicken.
As for the user-agent header, it's a standard http field, and can be used to identify the client (e.g., Chrome, etc.). The beta simply uses "PocketSense". If that doens't work, and if someone has a specific value that's needed, then I'll need to revisit.
Your willingness to address these rare issues is really appreciated. And I can't believe how quickly you put out a beta. Unfortunately, I tried it with Merrill without success. Here's the OFX file it returned: OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:0dca0c1c-f736-4b25-b985-7ffccd3e14d8
:OFX::SIGNONMSGSRSV1::SONRS::STATUS::CODE:2000:SEVERITY:ERROR:MESSAGE:Your request cannot be completed at this time, for assistance please contact us at 1.800.MERRILL (637.7455).:/STATUS::DTSERVER:20170410183148.913[-4:EDT]:LANGUAGE:ENG:FI::ORG:Merrill Lynch & Co. Inc.:FID:5550:/FI::/SONRS::/SIGNONMSGSRSV1::/OFX:
(In order to post the OFX file here I had to replace the left and right carat marks with : as the carats were being interpreted as HTML code).
I'll be interested to see if anyone else has better luck.
I edited rLib.py as indicated, ran it, checked to be sure rLib.pyc showed an updated time stamp. Then tested the Merrill account and, interestingly, the OFX file still showed SECURITY:NONE. I'm pretty sure I did everything correctly. I'm guessing Merrill isn't looking at that setting and just returns a canned response.
Since I don't use Merril, I can't really test. Another quick check would be to see what response you get using OFX 2.x. I started adding support when debugging the Discover issue. Set ofxV=211. I doubt this is the problem, but easy to check.
Robert, thanks for your efforts. I changed ofxver from 102 to 211 in my sites.dat file. The test didn't generate an error, but when I imported the file in Money I got the message: The file you attempted to import appears to be invalid or contains corrupt data....... I took a look and all I got was the header info (see below); there was nothing else in the OFX file.
OFXHEADER:100 DATA:OFXSGML VERSION:102 (I know it says 102 but it's 211 in sites.dat for Merrill) SECURITY:TYPE1 ENCODING:USASCII CHARSET:1252 COMPRESSION:NONE OLDFILEUID:NONE NEWFILEUID:NONE
I'm going to stick with just downloading the OFX file directly from the Merrill site. I don't have many transactions so don't need to do it very often. Thanks again for your help.
Hi Robert, Thank you very much for your great work! The add-on works great on discover and citi credit cards. There is an issue with chase bank account. It shows "The file you attempted to import appears to be invalid or contains corrupt data" in Money. I have enable the access in chase bank. Is there something wrong with the site.dat or something missed?
Is any else seeing this odd behavior with Discovercard?
Recently, I have been unable to d/l from Discovercard using the scripts. The error thrown was: 'Access Denied. You don't have permission to access "http..." on this server.'
I use the scripts with MS Money. I also use Moneydance. (I'm able to d/l Discovercard with Moneydance)
Looking at the Discovercard account in Moneydance, where '#' represents each digit of the card number, the credit card number syntax has changed from: '#### #### #### ####' (which represents the card number) to 'Discovr##X##X####X####'
That is not a typo. The beginning sequence is 'Discovr' (without the 'e') - the X's are just X's. Again, the '#' represent numbers.
What is particularly odd is that some of the numbers in the new syntax coincide with the card numbers and some do not and seem completely random.
e.g. The first 2 digits coincide with the 7th and 8th digits of the credit card. The second set of 2 digits does not coincide with any 2 digits of the credit card. The third set of 4 digits does not coincide with any 4 digits on the card. The last set of 4 digits coincides with the last 4 digits on the card.
I changed the account number in the scripts to the new syntax and numbers and I was then able to d/l from Discovercard.
I am getting Access Denied on Discover too after downloading the May 11 update. I use MS Money and PocketSense. I tried changing the syntax to what was posted above with the XXs and random numbers, but no luck.
What else can I try? Been at this for 3 hours now, so happy to have any suggestions. Net
Either go to the newer comments, or read my blog article at: https://microsoftmoneyoffline.wordpress.com/2017/05/12/discover-has-changed-your-account-id/
I've been using Pocketsense for several years. Love it! Yesterday I downloaded and installed your latest version (11-May-2017 18:50 EST). I can download stock and mutual fund prices but I'm having problems downloading OFX data from my credit cards and Fidelity accounts. I'm seeing the mesaage below. Any suggestions would be much appreciated.
** An ERROR occurred sending POST request to online.americanexpress.com Exception type : Exception val : endheaders() takes exactly 1 argument (2 given)
The ofx post header was modified for Discover, with the assumption that other banks wouldn't mind. Looks like you've disproved that assumption ;) I'll modify to support both header types.
Robert, I downloaded the beta zip and installed it like a typical, new version of PocketSense. I then used the Setup.py to test OFX retrieval from American Express. It failed with the following error message.
** An ERROR occurred sending POST request to online.americanexpress.com Exception type : Exception val : endheaders() takes exactly 1 argument (2 given) An online error occurred while testing the new account.
Could this be an error in my execution? Thanks -jim
I don't use AMEX, but configured a dummy account and tested using settings on ofxhome. It connected fine (i.e., made it through the failure point you're hitting without issue). It's either something odd about the account/username info, or your Python version.
Highly recommend updating to latest version of 2.7.x Python. I've used them all, but great success w/ the Anaconda distro.
Robert, I upgraded to Python 2.7.13 and that solved the problem. I reinstalled your your latest version (11-May-2017 18:50 EST) and everything is working again. Thanks for your quick responses and continued support of PocketSense.
Pocketsense has been working for me for several years now and I really appreciate all your work. Yesterday I hit a problem connecting with my Wells Fargo brokerage account. The request to connect is rejected with a message saying "sslv3 alert handshake failure". This problem has continued into today. Have Wells Fargo made changes to something? Thank You
Hi, You guys helped me with my Discover issue, so I am hoping that you can help me with Chase.I use pocketsense with MS Money Plus. I get the following error info on my xfr file:
/OFX//SIGNONMSGSRSV1//SONRS//STATUS//CODE/15510/SEVERITY/ERROR/MESSAGE/Please verify your identity within the next 7 days. Using your desktop computer, go to your bank’s website and visit the Secure Message Center for instructions.//STATUS//DTSERVER/20170528175341.231[-4:EDT]/LANGUAGE/ENG/FI//ORG/B1/FID/10898//FI///SONRS///SIGNONMSGSRSV1//CREDITCARDMSGSRSV1//CCSTMTTRNRS//TRNUID/2ddb49e0-6769-4c16-b1c8-58898052ddf0/STATUS//CODE/15500/SEVERITY/ERROR//STATUS//CLTCOOKIE/4//CCSTMTTRNRS///CREDITCARDMSGSRSV1///OFX/
Not sure what all that means, but I tried to go to my website which lets me login just fine, and look for a message in secure message, there isn't one. I do have two chase accounts to set up, myself and my husband, but I can't even get one to work anymore.
I am using this setup on my sites data: SiteName : CHASE AcctType : CCSTMT fiorg : B1 fid : 10898 url : https://ofx.chase.com ofxver : 103 bankid : brokerid : appid :QWIN appver :2400 mininterval
Any ideas, thoughts or pointers in the right direction would be appreciate. Net
Did the connection to Chase via the scripts work as expected in the past? OR Are you setting it up for the first time?
Have you enabled your account(s) for 'Linked apps and websites' at Chase.com?
Have you received an email from Chase which also states that you need to verify your identity?
You stated that you have two accounts @ Chase; have you tried logging on to Chase.com for the other account to see if the secure message center there has the reported message?
Kevin, Thank you for the quick reply. Walking through those steps was helpful. Chase was downloading just fine until May and I do have the financial software enabled on the Chase website. I had not thought to look at email, but today I checked and there was a message there. However, I did a search and had no previous emails from Chase. I checked secure messages and found the message there for the first time. It too was the first time one was listed. I do not know what changed and why they didn't appear before. But both accounts are now working fine now. Thank you!
Only account now that I do not have downloading is Barclaycard. Do you know if there is any info posted anywhere about that bank? I have to download it manually. I do not see it in the OFX directory.
I'm getting duplicated download transactions from Discover Card and Bank, which aren't matching with previously-downloaded items. Normally subsequent downloads don't exhibit this issue (even with overlapped date range). Thoughts?
Hi Derek. I wasn't having the issue a few days ago, but I'm not in a place where I can test at the moment. I'll check again in a few days, but I'm wondering what version of Python you're running? If it's not the latest 2.7.13, I'd recommend updating, as there seems to have been some issues w/ older versions.
Finally able to test, and don't see that issue. Do you see the following message during download? I guess I should also ask if you're using the Sunset edition of Money, or an earlier one. I only test on the Sunset version.
+Scrubber: Processing Discover Card statement.
Re Python versions, you're correct that the scripts are written for version 2.
Unfortunately, I can't replicate the problem. I'm assuming that we're only talking about transactions downloaded via the scripts, and that *none* have been manually downloaded/imported from Discover? The transaction ID values assigned by the script downloads will never match what Discover provides natively via their manual (web site) download option.
I thought I only downloaded via scripts but realized I may have manually downloaded/imported from Discover when obtaining their new acct # format. If that was the case, perhaps all subsequent downloads via scripts would cause the dups, but future transactions should no longer exhibit this?
Derek - all downloads via scripts should work properly, but as Robert stated, those transactions you downloaded from the website will repeat when the script obtains those. Probably best to delete the web downloaded transactions as you find the repeats OR shorten the date range.
Apparently, there is a Wells Fargo OFX connection issue in progress. Pocketsense has not been able to connect since the weekend. Here is more from the Moneydance forum:
Thanks Kevin. No, this did not resolve the error. BTW, this is the message: WELLS FARGO : xxxxxxxxxxx : Getting records since: 20170522 ** An ERROR occurred sending POST request to ofxdc.wellsfargo.com Exception type : Exception val : [Errno 1] _ssl.c:499: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
From what I've been reading on this matter, the general consensus is that WF now requires two-factor login and *that* may be causing the sudden errors.
Ok, I resolved the issue. I did some digging around with the ssl error. So, Wells Fargo must have upgraded the SSL services on the ofx server. So, I found this article: https://stackoverflow.com/questions/35405092/sslerror-sslv3-alert-handshake-failure
Sure enough, I was running Python 2.7.8, and only Python 2.7.9 and later has the SSL feature updates (SNI is the feature). So, I would just load the latest 2.7.x Python if you see this condition. And, you have to add the following to the site.date for WF Entry: ofxVer : 103
Good Motning, Pocketsense has been working for me for several years now and I really appreciate all your work. This morning I have hit problems with all my Schwab Brokerage accounts which caused my regular we login to get suspended. The pertinent messages in the OFX file shows:-
"CODE" 15500
"SEVERITY"ERROR
"MESSAGE"Signon (for example, user ID or password) invalid (ERROR)
I got my password reset and was able to access the accounts through the web-site, but when I changed the password in Pocketsense and retried the scripts the problem was still there. I'm not sure if Schwab has made changes to their OFX interface or whether it's due to the holiday? Anyone else hitting this issue? Thanks & Happy 4th! Thank You
Anyone else having issues downloading from Discover? Just got an error today saying "You don't have permission to access "http:\\ofx.discovercard.com\" on this server. Not sure if this is a temporary glitch, or a change in protocol. Will watch for a bit before acting...
ReplyDeleteI've been having problems with Discover all this week.... I am getting an HTTP error 403, which is a "forbidden" error, and not being able to connect...
ReplyDeleteHarold
I posted the same comment on my blog too. I haven't had any success by calling their customer service web support. It is difficult to get routed to the right tech support person that can resolve the server issue. I tried claiming I use Quicken, and they say it's their problem. Then I tried saying I use Microsoft Money and they say "we don't support that program". Looking at a QFX file from web downloads doesn't offer much help but it does list the FID as 9625 instead of 7101.
ReplyDeleteFrom the Moneydance forum at
ReplyDeletehttp://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working
Quoting:
Posted by robf on Feb 24, 2017 @ 11:03 AM
robf's Avatar
I HAVE THE SOLUTION!
The HTTP request User Agent header has changed. In my code (I'm not using Moneydance but my own creation), the Discover server used to accept this header: httpReqst.UserAgent = "httpclient"
Now, if I comment out that header or set httpReqst.UserAgent =null I can download from Discover; if it's left uncommented I get the 403 Forbidden error
The Moneydance software guys (and gals) should update their OFX request with this change for Discover. Be warned, other institutions, such as TD Bank, require the UserAgent header to be set.
Sounds promising. I'll look into it for the scripts... hopefully this weekend. Thanks Andrew!
DeleteHi Robert,
DeleteThank you for your attention to this matter.
Here are some direct links to the posts from robf on the Moneydance forums regarding his findings with Discover.
http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working#comment_42026919
http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working#comment_42027042
http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working#comment_42027064
HTH -Kevin N.
Over at ofxhome, robler posted at
Deletehttp://www.ofxhome.com/ofxforum/viewtopic.php?id=47793
Quoting:
Here's the full HTTP header list that works for Discover:
httpReqst.ContentLength = enc.Length;
httpReqst.Method = "POST";
httpReqst.ContentType = "application/x-ofx";
httpReqst.UserAgent = null;
httpReqst.ServicePoint.Expect100Continue = false;
The first 3 are necessary for OFX requests at any institution. The last 2 are Discover specific, meaning other institutions might require other values.
End Quoting.
I don't know if robf and robler are the same person. According to MSDN, the default Expect100Continue value is true.
I am glad to see I am not the only one having issues - this stared about 7 - 10 days ago for me but my error is slightly different:
ReplyDelete** An ERROR occurred sending POST request to ofx.discovercard.com
Exception type:
Exception Val : {Errno 10061} No connection could be made because the target machine actively refused it.
As others have in this thread, I did reach out to Discover but they only provided assistance for manual download/import.
On a side note, I cannot express how thankful I am to have this site and your help and input in keeping Microsoft Money alive for us Robert! Thank you so much!
Hi, i've absolutely had it with the crap called quicken 2011 & 2014 and determined not to buy 2017. I really don't trust it to add a column of numbers correctly (i've got evidence). Would anyone know a way to export data out of Quicken into Money keeping the categories & data? Got an aweful lot over these 6 years. thanks
ReplyDeleteYou may have the most success by removing most of Money's categories in a blank data file and then setting up new categories in the same fashion as your Quicken has them prior to importing the QIF file created by Quicken. I'm not sure if you should setup the accounts prior to the import or if Money will do that for you.
ReplyDeleteThe expert on the subject is Dan (support at gaiersoftware.com) as he wrote the converter for Money to Quicken back in 2009.
Let me know of your experience and I'll include it in my blog: https://microsoftmoneyoffline.wordpress.com/
- ameridan
Just FYI... but my weekend turned out to be *busy*, so I haven't had time to dig into the Discover issue yet.
ReplyDeleteIt seems BECU has gone through some system changes and now users are unable to connect.
ReplyDeletehttp://help.infinitekind.com/discussions/online-banking/5046-boeing-employees-credit-union-aka-becu-has-new-ofx-url has an answer from BECU, but doesn't seem to help.
Tried various combinations and could not get it to work. I've messaged BECU again to try to get the correct url. They've changed this a couple of times in the past and it has been difficult to get the correct path. It appears they are using ofxver 102, so that should not be the issue.
DeleteKR
Just fyi... but manual downloads seem to always be ofxver 102, but direct-connect may not be.
DeleteMore info on it here. http://www.ofxhome.com/ofxforum/viewtopic.php?id=47793
ReplyDeleteYes... I actually sent a request to @robler, looking for the same info being requested there. I have a few ideas, but it will be more involved to test than I've had time to try... yet.
DeleteGot an answer from BECU, I haven't tried any changes yet:
DeleteThank you for contacting us. I’m sorry you’re having trouble connecting to BECU from your external money management program. I believe I can help.
Following the last update to the BECU website last weekend we’ve received reports from other members that connections to their third-party accounting programs were interrupted during the update. To fix this, you should reset the connection. How this is done varies from one program to another, so if you need assistance I recommend you contact the maker of the program you use and ask them to reset the connection to BECU. Make sure they use the following information to download transactions:
Org Name: BECU
OFX ID: 1001
URL: https://onlinebanking.becu.org
If you have additional questions you can reply to this message. You can also contact us by phone during our business hours of Monday Friday 7:00 AM - 7:00 PM PST and Saturday 9:00 AM - 1:00 PM PST at 800-233-2328.
Sincerely,
Jeremy Nelson
BECU Technical Support Analyst
Phone 800-233-2328
If the above settings don't do the trick... if you use ofxVer: 103 for BECU, then you may need to delete the CLIENTUID line at the bottom of your sites.dat file. That's the only persistent "setting" kept by the scripts, other than the site and account settings.
DeleteBECU systems changed, but watching what Quicken does led me to these settings, which work.
DeleteSiteName : BECU
AcctType : BASTMT
fiorg : becu
fid : 325081403
url : https://onlinebanking.becu.org/ofx/ofxprocessor.asp
bankid : 325081403
brokerid :
appid : QWIN
appver : 2300
mininterval:
ofxver : 103
SiteName : BECU CC
AcctType : CCSTMT
fiorg : becu
fid : 3670
url : https://onlinebanking.becu.org/ofx/ofxprocessor.asp
bankid : 325081403
brokerid :
appid : QWIN
appver : 2300
mininterval:
ofxver : 103
Marvelous! I just saw this info on the Money dance site, and they mentioned using pocket sense to connect, too!
DeleteKR
Just posting this here for reference, so I don't forget. Another (related) post from another site. See https://getsatisfaction.com/quickencommunity/topics/cant-download-discover-transactions-to-mac-quicken-2007-due-to-ol-226-error
ReplyDelete"I just got off chat with Discover. They stopped supporting Quicken 2007 as of Jan 2017. They then made some changes in mid-February 2017 (probably Feb 18 as Michael Smith mentioned earlier in this thread) which now keeps Quicken 2007 from working with Discover ... "
That was QWIN ver 1600 which I would have thought stopped working long ago. I think the Pocketsense default version is around 2300, so I don't think that is the issue at all. I've actually tried 2500 as well.
ReplyDeleteI have been playing with ofx.py.
ReplyDeleteerrmsg= "** An ERROR occurred attempting HTTPS connection to"
h = httplib.HTTPSConnection(host, timeout=5)
h.set_debuglevel(1)
h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1)
h.putheader('Host', host)
h.putheader('Content-Length', str(len(query)))
h.putheader('Content-Type', 'application/x-ofx')
h.putheader('Accept', '*/*, application/x-ofx')
h.putheader('Accept-Encoding', 'identity')
errmsg= "** An ERROR occurred sending POST request to"
p = h.endheaders(query)
# p = h.request('POST', selector, query,
# {"Content-Type": "application/x-ofx",
# "Accept": "*/*, application/x-ofx"
# }
# )
errmsg= "** An ERROR occurred retrieving POST response from"
#allow up to 30 secs for the server response (it has to assemble the statement)
h.sock.settimeout(30)
response = h.getresponse().read()
f = file(name,"w")
f.write(response)
f.close()
Running this version of ofx.py in Setup.py option 7, test mode, only for my Discover accounts yields (private info rewdacted)
DISCOVER BANK : ********** : Getting records since: 20170131
send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 680\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\nOFXHEADER:100\r\nDATA:OFXSGML\r\nVERSION:102\r\nSECURITY:NONE\r\nENCODING:USASCII\r\nCHARSET:1252\r\nCOMPRESSION:NONE\r\nOLDFILEUID:NONE\r\nNEWFILEUID:************************************\r\n\r\n\n\n\n20170303103349\n**PRIVATE**\n**PRIVATE**\nENG\n\nDiscover Bank\n12610\n\nQWIN\n2500\n\n\n\n\n\n************************************\n4\n\n\n031100649\n**********\nCHECKING\n\n\n20170131\nY\n\n\n\n\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Server: AkamaiGHost
header: Mime-Version: 1.0
header: Content-Type: text/html
header: Content-Length: 269
header: Expires: Fri, 03 Mar 2017 15:33:50 GMT
header: Date: Fri, 03 Mar 2017 15:33:50 GMT
header: Connection: close
header: Set-Cookie: ...
Invalid OFX statement.
** Review .\xfr\DISCOVERBANK20170303103349331974.ofx for possible clues...
An online error occurred while testing the new account.
Note, NO User-Agent header, no Expect header. Still get 403 response. These are the same headers sent by the original code, slightly different order.
Here is hard-coding the query to match the one posted by robler on infinitekind. I hard-coded my user name, password, and account number in the query (redacted in output) as query = '...' just before the try:.
DISCOVER : **************** : Getting records since: 20170131
send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 720\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\n\r\n\r\n\r\n\r\n20170228232807\r\n**PRIVATE**\r\n**PRIVATE**\r\nENG\r\n\r\nDiscover Financial Services\r\n7101\r\n\r\nQWIN\r\n2500\r\n\r\n\r\n\r\n\r\n0\r\n\r\n\r\n**PRIVATE**\r\n\r\n\r\n20130101000000\r\n20170228232807\r\nY\r\n\r\n\r\n\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Server: AkamaiGHost
header: Mime-Version: 1.0
header: Content-Type: text/html
header: Content-Length: 269
header: Expires: Fri, 03 Mar 2017 16:04:13 GMT
header: Date: Fri, 03 Mar 2017 16:04:13 GMT
header: Connection: close
header: Set-Cookie: ...
Invalid OFX statement.
** Review .\xfr\DISCOVER20170303110412992001.ofx for possible clues...
An online error occurred while testing the new account.
Sending his working OFX query still yields a 403.
Until robler can post his/her header's, I don't think we will get anyplace.
Formatting total messed up in second test. Changed all less than ... greater than to {...} so board doesn't strip them.
DeleteDISCOVER : **************** : Getting records since: 20170131
send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 720\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\n{?OFX OFXHEADER="200" VERSION="211" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="NONE"?}\r\n{OFX}\r\n{SIGNONMSGSRQV1}\r\n{SONRQ}\r\n{DTCLIENT}20170228232807{/DTCLIENT}\r\n{USERID}**PRIVATE**{/USERID}\r\n{USERPASS}**PRIVATE**{/USERPASS}\r\n{LANGUAGE}ENG{/LANGUAGE}\r\n{FI}\r\n{ORG}Discover Financial Services{/ORG}\r\n{FID}7101{/FID}\r\n{/FI}\r\n{APPID}QWIN{/APPID}\r\n{APPVER}2500{/APPVER}\r\n{/SONRQ}\r\n{/SIGNONMSGSRQV1}\r\n{CREDITCARDMSGSRQV1}\r\n{CCSTMTTRNRQ}\r\n{TRNUID}0{/TRNUID}\r\n{CCSTMTRQ}\r\n{CCACCTFROM}\r\n{ACCTID}**PRIVATE**{/ACCTID}\r\n{/CCACCTFROM}\r\n{INCTRAN}\r\n{DTSTART}20130101000000{/DTSTART}\r\n{DTEND}20170228232807{/DTEND}\r\n{INCLUDE}Y{/INCLUDE}\r\n{/INCTRAN}\r\n{/CCSTMTRQ}\r\n{/CCSTMTTRNRQ}\r\n{/CREDITCARDMSGSRQV1}\r\n{/OFX}'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Server: AkamaiGHost
header: Mime-Version: 1.0
header: Content-Type: text/html
header: Content-Length: 269
header: Expires: Fri, 03 Mar 2017 16:04:13 GMT
header: Date: Fri, 03 Mar 2017 16:04:13 GMT
header: Connection: close
header: Set-Cookie: ...
Invalid OFX statement.
** Review .\xfr\DISCOVER20170303110412992001.ofx for possible clues...
An online error occurred while testing the new account.
Just fyi... but I did ~ the same tests yesterday, with the same results.
DeleteYes, that's a total bummer. Doesn't look easy now.
DeleteFinally had a little time yesterday to actually focus on the Discover issue. I cobbled together a small .net module to test connections using their http library. I was able to get a successful connection and ofx response, but I tested using OFX 2.x (xml), so still need to test with sgml. Initial (quick) comparisons between the python httplib debug output vs .net output didn't reveal anything obvious, so more to do...
ReplyDeleteThis was posted by pgs over on ofxhome yesterday.
ReplyDeleteQuoting:
I was hitting the same thing, but thanks to robler's notes, I was able to get it working again as well. The server is very finicky about the order of the HTTP headers -- this sequence works for me:
POST / HTTP/1.1
Content-Type: application/x-ofx
Host: ofx.discovercard.com
Content-Length: [the length of the request]
Connection: Keep-Alive
With those headers, the XML request robler posted above works for me. Hope this helps.
End Quote
I was able to confirm that this order works in pocketsense for Discover Bank for both bank accounts and credit cards. This is still sending the normal ofxsgml headers. I made this change in ofx.py in function doQuery().
***** CAUTION: THIS HAS NOT BEEN FULLY TESTED WITH OTHER BANKS *****
Let Robert supply a tested version that he confirms works with other banks also.
*********************************************************************
try:
errmsg= "** An ERROR occurred attempting HTTPS connection to"
h = httplib.HTTPSConnection(host, timeout=5)
# Uncomment next line to turn on debugging info
# h.set_debuglevel(1)
h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1)
h.putheader('Content-Type', 'application/x-ofx')
h.putheader('Host', host)
h.putheader('Content-Length', str(len(query)))
h.putheader('Connection', 'Keep-Alive')
errmsg= "** An ERROR occurred sending POST request to"
p = h.endheaders(query)
# original code
# p = h.request('POST', selector, query,
# {"Content-Type": "application/x-ofx",
# "Accept": "*/*, application/x-ofx"
# }
# )
errmsg= "** An ERROR occurred retrieving POST response from"
#allow up to 30 secs for the server response (it has to assemble the statement)
h.sock.settimeout(30)
response = h.getresponse().read()
f = file(name,"w")
f.write(response)
f.close()
Excellent. I actually went down a different path when I started focusing on the issue. Discover appeared to be fully supporting OFX 2.x for direct-connect, so I assumed it would eventually become the "standard". That may not happen, but I started adding support for OFX 2.x messages (xml), and got that working. The catch is that Money doesn't support OFX 2.x, *but* the difference between sgml and xml is minor. Replacing the message header with a 1.x version gets past the Money import "format check", and the extra trailing field closures are accepted (although optional) in sgml. Seems to work fine.
DeleteI had assumed that I'd either be adding a compiled module for the http connection, or writing a sockets interface in the scripts (for full control). If the manual httplib request you've used works reliably, then that's even better!
Thanks Andrew.
Ditto Andrew - I made 2 setup.py/ofx.py combos for Discover/non-Discover accounts and am back in business (I prefer downloading per-account rather than in a batch). MUCH appreciated for both of y'all's work!
DeleteFrom another post by pgs:
DeleteQuoting:
The key was robler's comment about the required options for the .NET HttpWebRequest class -- once I got that working I was able to log the HTTP request. It's so sensitive that I wasn't even able to get the http library I was previously using to work and ended up writing a custom client just for this server.
End quoting
pgs was able to see the debug the http headers in .net and found the order that worked. This despite the standard stating the header order should not matter for unique headers!
I took a look at the generated ofx responses.
Discover Card is still broken the same like {FITID}FITID20170216408.850 after scrubbing.
Discover Bank is still broken the same like {FITID}SDF7957870 after scrubbing.
Why is it that Discover is just so incompetent.
**** CAUTION ****
DeleteI strongly urge that if you don't need to download Discover transactions today, you should wait until Robert has an official release.
**** CAUTION ****
NOTE: The previous post was deleted because a debugging line that might change the behavior was left inadvertently.
This should be a safer change for the doQuery() function in ofx.py. It explicitly treats Discover stuff separately. Note that if you want to try it, you need to correct the indentation which is stripped by the board.
def doQuery(self,query,name):
# urllib doesn't honor user Content-type, use urllib2
url = FieldVal(self.site,"url")
isDiscover = 0
if 'DISCOVERCARD' in url.upper():
isDiscover = 1
garbage, path = urllib2.splittype(url)
host, selector = urllib2.splithost(path)
response=False
try:
errmsg= "** An ERROR occurred attempting HTTPS connection to"
h = httplib.HTTPSConnection(host, timeout=5)
# h.set_debuglevel(1)
errmsg= "** An ERROR occurred sending POST request to"
if isDiscover:
h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1)
h.putheader('Content-Type', 'application/x-ofx')
h.putheader('Host', host)
h.putheader('Content-Length', str(len(query)))
# h.putheader('Accept', '*/*, application/x-ofx')
# h.putheader('Accept-Encoding', 'identity')
h.putheader('Connection', 'Keep-Alive')
p = h.endheaders(query)
else:
p = h.request('POST', selector, query,
{"Content-Type": "application/x-ofx",
"Accept": "*/*, application/x-ofx"
}
)
errmsg= "** An ERROR occurred retrieving POST response from"
#allow up to 30 secs for the server response (it has to assemble the statement)
h.sock.settimeout(30)
response = h.getresponse().read()
f = file(name,"w")
f.write(response)
f.close()
except Exception as inst:
self.status = False
print errmsg, host
print " Exception type:", type(inst)
print " Exception Val :", inst
if response:
print " HTTPS ResponseCode :", response.status
print " HTTPS ResponseReason:", response.reason
if h: h.close()
I was late getting home, but had time to test the updates done thus far. In a nutshell, the new version builds the ofx header using the rules posted by Andrew above, and worked with all of the sites that I personally use. Additionally, I added support for OFX 2.x, and successfully tested with Discover. Actually, I tested Discover with ofxVer=102, 103, and 211, and all worked fine w/ Money. Discover bank downloads appear to be offline at the moment, so I couldn't test import to Money, but connections worked fine earlier when I first wrapped up the script edits.
ReplyDeleteI've posted a beta version @ https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip . If anyone has time to test using their account settings, that would be helpful. I'd recommend stepping through each individually using the Setup module, rather than all at once with Getdata.
All of mine worked flawlessly, including Discover Card and Bank:
DeleteTDAMERITRADE
USAA FEDERAL SAVINGS BANK (Checking)
USAA FEDERAL SAVINGS BANK (Savings)
USAA CC (Amex)
E-TRADE INVESTMENTS
OPTIONSXPRESS
DISCOVERBANK
DISCOVERCARD
CHASE CC
CITIBANK COSTCO
Thanks so much!
I have been using Pocket Sense ever since Money abandoned us. If I am techie at all it would be level 1, but I somehow managed to set it up reading and rereading instruction. I have not a clue what most of this discussion means, but I knew Discover screwed it up and wouldn't help, however I took a chance and downloaded the beta file and managed to set it up again (a long time since I set it up first time). It works with my Citi and my Discover. I don't know why, but thank you, thank you very much is all I can say. Rette
DeleteHi Robert ... Just tested your latest zip and it works fine on all my Institutions just like the current version does.
ReplyDeleteOne new wrinkle just showed up today.....Citibank Fails with INVALID OFX both current version AND new beta version. I am able to download with win phone version of my software, but not pocketsense. I suspect something Citi changed over the weekend as everything worked before Monday morning.
Looks like banks are making changes to keep us occupied.
Raymond
Scratch my previous message about Citibank....It also works fine with the beta version of pocketsense. The current version does not work.
ReplyDeleteRaymond
Using the current non-beta version, I am also now failing with a 403 error with Citibank.
DeleteHarold
Robert,
ReplyDeleteAny chance you can fold in the siteClientUID changes previously proposed as well? I've been running Pocketsense with those changes manually added just in case Schwab ever switches to ofx 1.03 and I have to keep manually adding the changes in for each revision. It's a nice revision for those that have multiple accounts at one ofx 1.03 institution.
thanx,
ameridan
--------------------------------------------------------------------------------
--- OfxPy/orig/ofx.py Wed Feb 11 10:07:36 2015
+++ OfxPy/ofx.py Thu Feb 11 23:37:36 2016
@@ -86,7 +86,14 @@
clientuid=""
if "103" in self.ofxver:
#include clientuid field only if version=103, otherwise the server may reject the request
- clientuid = OfxField("CLIENTUID",userdat.clientuid)
+
+ # if a site-level clientUID is defined, use it, otherwise default to global clientUID
+ clientuidval = userdat.clientuid
+ if FieldVal(site,"siteClientUID") <> '':
+ clientuidval = FieldVal(site,"siteClientUID")
+
+ clientuid = OfxField("CLIENTUID",clientuidval)
+ print "using ClientUID of: ", clientuidval
fidata = [OfxField("ORG",FieldVal(site,"fiorg"))]
fidata += [OfxField("FID",FieldVal(site,"fid"))]
--- OfxPy/orig/site_cfg.py Wed Feb 19 20:51:56 2014
+++ OfxPy/site_cfg.py Thu Feb 11 23:11:21 2016
@@ -142,6 +142,7 @@
bankid=''
brokerid=''
ofxver = '102'
+ siteClientUID = ''
appid = DefaultAppID #defined in control2.py
appver = DefaultAppVer
mininterval = 0
@@ -161,6 +162,7 @@
'BANKID': bankid,
'BROKERID': brokerid,
'OFXVER': ofxver,
+ 'SITECLIENTUID': siteClientUID,
'APPID': appid,
'APPVER': appver,
'MININTERVAL': mininterval,
@@ -181,6 +183,7 @@
elif field == 'BANKID': bankid = value
elif field == 'BROKERID': brokerid = value
elif field == 'OFXVER': ofxver = value
+ elif field == 'SITECLIENTUID': siteClientUID = value
elif field == 'APPID': appid = value
elif field == 'APPVER': appver = value
elif field == 'MININTERVAL': mininterval = int(value)
Hi Dan: Can you refresh my memory on this one? I recall adding support for multiple accounts at one institution, where they all have the same username. Is this related, or different?
ReplyDeleteI think Dan may be referring to the issue that came up at Chase that forces us to run two instances of Pocketsense because of some GUID (or something like that) field?
DeleteHarold
This is that issue, but I don't recall you actually incorporating it. Here's the link to the author's site to jar your memory: http://pastebin.com/9UJ333RZ
ReplyDeleteDo you think ofx 2.0 will be next for some of these financial institutions, rather than 1.03? If so, I guess we could hold off till it actually happens, but I thought the author (forget who) had come up with a nice solution.
------------------------------------------
By the way I just tested your Beta at Schwab, and Money didn't like it "The file you attempted to import appears to be invalid..."
Here's my output file header:
OFXHEADER:100
DATA:OFXSGML
VERSION:2500
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:dd1a870f-23a5-47dc-bae7-30ec160c1139
vs. my file header from Friday:
OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
where did version 2500 come from?
Raymond, my Citibank stopped working today too. I got rid of my beta before trying because it didn't work for Schwab. What are these banks doing?
ReplyDeleteI set the default version to 2500... to see if it "broke" anything. Guess it did :) I wondered if servers did a "greater than or equal to", or looked for specific values. Try Schwaab w/ a lower value, and let me know.
ReplyDeleteThe beta worked for Schwab; it's Money that didn't like the file and that was the only difference I could find, But I substituted the entire header in today's ofx file and Money still rejected it, so the version wasn't the problem.
DeleteThe non-beta download works for Schwab? If so, will need to compare the message body, since I don't think I changed anything in the request, except when ofxVer=2xx.
DeleteRobert,
DeleteYou have ofxver set to 2400. I had 2500 in my sites.dat file with my current setup and it works.
I think I'm confused. I set default appVer=2400, but ofxVer should default to 102 if not specified in sites.dat. I missed that above (appVer vs ofxVer).
DeleteDoes Schwab work w/ the beta when ofxVer=102 and appVer=2500?
Excellent catch Robert!!
DeleteI apologize ( I was confused too seeing VERSION:2500). I was experimenting in December and mistakenly entered 2500 in my sites.dat file for Schwab ofxVer, rather than appVer. Until the beta, it was ignored I guess. All is well now with Schwab and your beta. YEAHHH
And Citi and Discover and PNC all work too! Great job.
DeleteThat's a relief :) Had me worried. Thanks Dan!
DeleteI found the discussion re: possibly adding site-specific ClientUID values, and remember why I didn't do anything. It's one of those "easy to do but messy" hacks. If I'm understanding, the underlying issue is that each username needed it's own UUID, or more likely, the bank is associating a UUID with a username, and doesn't allow it to be reused. I tossed out the idea of adding a site-specific ClientUID field, so a user can have multiple copies of a site, and use one for each unique username. It was an easy fix from a code perspective, but messy to use and not well understood, so I left it alone. What I hoped to learn (but didn't) was whether a site required a username to have the same UUID across all accounts, or if it was ok to assign unique UUIDs per account setting. That would be an automatic (elegant) solution.
ReplyDeleteHi All,
ReplyDeleteI just tried the beta with Discover Credit Card.
I'm getting the following error when testing it with Setup.py:
** An ERROR occurred sending POST request to ofx.discovercard.com
Exception type:
Exception Val : endheaders() takes exactly 1 argument (2 given)
An online error occurred while testing the new account.
-Kevin N.
The board remove the text after the Exception type: probably due to the less than sign and the greater than sign.
DeleteException type:(less than sign) type 'exceptions.TypeError' (greater than sign)
Kevin, did you overwrite control2.py, ofx.py and rlib.py? I got that message thinking that all the changes were in ofx.py but apparently the other 2 files are needed too.
ReplyDeleteHi NexTek,
ReplyDeleteI extracted the new beta zip file to a new empty folder.
Ran Setup.py
Closed Setup.py
Ran Setup.py again
Added my Discover card account and tested it.
That's when I got the error
-Kevin N.
I replicated your steps, and it worked for me. Before tracing out differences, what version of Python are you running? I tested using Python 2.7.12 and 2.7.13.
DeleteHi Robert,
DeleteI'm using Active Python 2.6.5.12
It looks like 2.7.13 is available at Active, I'll install that and give it a try.
Thanks for getting back to me. I'll let you know how it goes.
-Kevin N.
Hi Robert,
Delete2.7.13 did the trick! Discover Card works as expected.
Thank you.
-Kevin N.
Hi NexTek,
ReplyDeleteHave you tried John's changes to ofx.py and site_cfg.py for separate ClientUID's with the new beta?
-Kevin N.
No, I don't actually have a way to test it; I just added it to my previous version thinking that soon ofx 1.03 would be implemented by more financial houses. Like Robert implied, it is easy for someone that has a need for it to add, rather than having separate Pocketsense implementations for each additional account.
DeleteHi NxtTek,
DeleteOK. I need the changes for His and Her's Chase credit card accounts. I'll give them a whirl and see how it works.
-Kevin N.
I kinda' wish Robert would add the code, as it doesn't get in the way at all if not needed. Unless he has a more "elegant" solution. ;)
DeleteHere's my blog article on implementing it for others to reference. https://microsoftmoneyoffline.wordpress.com/2016/04/26/resolving-errors-for-those-having-multiple-accounts-at-chase-and-other-ofx-103-banks/
I don't mind adding it, and it's easy to do. I originally had thoughts of automating, creating unique uuid values for each client/username/account combo, but wasn't sure if the uuid should be the same for multiple accounts under the same username? The idea was to create a (hashed) lookup table of unique pairs, adding new ones as they come along. It wouldn't be human "readable", but there wouldn't be any extra config.
DeleteHi Robert, everyone -
ReplyDeleteI downloaded and setup the beta version but am still receiving the same error. I did install Python 2.7.13 (from 2.7.5) and still no luck:
** An ERROR occurred sending POST request to ofx.discovercard.com
Exception type: (less than sign)class 'socket.error'(greater than sign)
Exception Val : {Errno 10061} No connection could be made because the target machine actively refused it.
Ben
Looks like you have a different issue. Compare your sites.dat entry for Discover to the following:
DeleteSiteName : DISCOVER
AcctType : CCSTMT #credit card
fiorg : Discover Financial Services
fid : 7101
url : https://ofx.discovercard.com
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval:
timeOffset :
Thanks for the follow-up Robert! My sites.dat entry was missing the ofxVer: 103 line - I added it but the error persists. :(
DeleteIs it only happening with Discover (i.e., do other sites work)?
DeleteRobert, I'll keep the ofxVer in mind, but mine worked today leaving that blank.
DeleteYou are correct... but I knew the settings above worked on mine. I tested with 102, 103, and 211 yesterday.
DeleteHi Robert - was out of town the last couple of days so I apologize for the delay in my response. To answer your question, yes, it only happens with Discover - I have it setup for AMEX and it works fine.
DeleteHi Robert - out of curiosity, I tried 102 as the value for ofxVer and it works! Wahoo! Thank you!
DeleteHi Robert,
ReplyDeleteI just want to confirm that the new beta works with:
Chase Credit Cards - 2 accounts (with John's changes)
Discovercard - 2 accounts (with ofxVer 103)
Regions Bank - 4 accounts
Fidelity Investments - 4 accounts
Thank you again for your continued work on the scripts.
-Kevin N.
Regarding the use use "user specific" ClientUID values, before adding the site->clientUID option, I'd like to understand what we're trying to fix. I know that some folks are using a custom UID value for each username (his and hers, if you will), but which of the following requirements is true:
ReplyDeletea) a unique UID value is needed for each site+username combo
b) a unique UID value is needed for each site+account combo, or
c) a unique UID value is needed for each site+username+account combo
Anyone know?
I believe that for Chase, it's site+username.
DeleteHarold
Hi Robert,
DeleteIn my case with Chase Credit Cards (his and her accounts) Obviously, the site is the same, the account numbers are different and thus the usernames and passwords are different.
The (hers) account uses a unique siteClientUID.
The (his) account uses the global ClientUID
Both use ofxVer 103
I retained the siteClientUID and the global ClientUID from my prior version of the scripts. In the past, changing the ClientUID triggered a prompt from Chase to verify my ID at their website. I'm thinking that a new or different ClientUID appears as a new computer / connection and they need to add that connection as a trusted connection.
HTH -Kevin N.
Since Fidelity (and maybe others) allow the same account number for mulitple usernames, I'm going to say that keying to the site and username is the safest bet. My preference at this point will be to simply add unique clientUID values based on site+username, and store them in a lookup table/file. The site and username would be hashed, for security. This approach would be fully automatic, with nothing special by the user. If it turns out we need site+user+account lookup, then it's easy to add. Sound ok?
DeleteSounds good to me Robert. Sounds like Kevin may have to make another verification call for their Chase accounts (as an example) unless the table allows overriding the automatically generated clientUID values with their existing ones.
ReplyDeleteI wondered about that. Kevin: Do they require verbal confirmation, or is just an email?
DeleteThis was a while ago now, but I think that Chase just requires a different UID value...
DeleteHarold
Hi Robert,
ReplyDeleteIt seems that when Chase detects a new computer / connection it triggers a notification email that requires the user to log onto the Chase website. Then, through a secure message, the user clicks on a 'Verify your identity' link.
My only concern is that they [Chase] may limit the number of trusted connections to any given account. I'll look further into that on their site.
-Kevin N.
Hi Robert,
DeleteI sent a message to Chase regarding the number of trusted connections / computers that can be assigned to any given account.
I won't bore you with the somewhat boilerplate response. Essentially, all that is needed is to verify my identity with each new computer / connection / ClientUID.
If I understand you correctly, your plans are to create a separate table for ClientUIDs. Can that table remain intact with subsequent upgrades to the scripts? If so, then that would be ideal.
-Kevin N.
My intent was that the ClientUID values would be reused indefinitely for a matching url + username.
DeleteBrilliant! Thank you for your consideration.
DeleteJust downloaded the beta. It seems like Discover worked fine, as did every other account including Chase.
ReplyDeleteRobert, Looking at the Beta reminded me of two scrubber issues.
ReplyDeleteThe first had to do with Discover and the strange $0.00 "second line in statement" transactions that it downloads. Did you ever come up with a scrubber for that? I remember reading about it years ago.
Second we had discovered that some 401k downloads (Net Benefits, for instance) would put the wrong sign in front of reinvestment transactions, and Money would ignore the transaction. I wrote a quick scrubber for it a while back. Are you interested in incorporating it to the main trunk?
Carlos
1. I had decided not to ignore the $0.00 transactions, since they usually contain additional transaction info. I'm not opposed to adding a filter option, but that was how I'd left it before.
Delete2. Pretty sure the "wrong sign" issue has been addressed in scrubber.py (?).
Hmm. Most of my $0.00 transactions usually have some random, 11-character, meaningless code. Not a big deal either way. I could see how sometimes the $0.00 transaction may be useful
DeleteThe version of scrubber.py I have shows a change by you from 2013 that addresses INVBUY or INVSELL transactions. From the comments I made in the file year+ ago, it looks like I copied your_scrubINVsign() functions to create a new function that fixes REINVEST transactions.
And once again, thanks for all the work you've done with this. It's amazing we are still able to use MS$ eight years after it went off the market.
Should have said "some" have extra info, but you're right. Most do not. I may play around with it.
DeleteJust to clarify, we're only talking about adding transactions with a REINVEST type?
Robert,
DeleteSorry, I wasn't checking in for a while.
On Discover: yes, some transactions have extra info, and many do not. Contrary to what NxtTek say, these are not pre-authorizations, as they can show up days after the related transaction. They are some "extra" info that Discover includes in their statements, like cash back, but for some reason when you download they just show up as a meaningless code.
On Fidelity: Yes, I have an equivalent fix for REINVEST transactions to the one you have for INVBUY and INVSELL.
"$0.00 transactions usually have some random meaningless code."
ReplyDeleteThese are pre-authorization transactions, meaning a real transaction will probably be occurring the next day. Since I download on a daily basis, their existence can be informative for fraud detection if you haven't ordered anything.
On Discover, I've found that it sends a $0.00 transaction only when it is a transaction that is eligible for their increased cashback offer for that quarter. Citibank occasionally sends a $0.00 transaction, but it is usually for something like a "membership charge" which my account doesn't actually get charged. None of my other banks sends $0.00 charges.
DeleteHarold
Hi, Robert... if you're considering including minor improvements, the change that I've made locally that I find useful is retrying a bad HTTP return code 5 times. For each time, I put out a line telling what the return code I got was. (Even with this 403 thing that is coming back now from Discover and Citibank, it retried five times before giving up.) I can't tell you why, but various banks occasionally come back with a bad HTTP return code, and just retrying it makes it work fine. Chase does this about once a month, and other banks seem to do it, too, from time to time.
ReplyDeleteI'd be ok with specific retry reasons, but there are legitimate causes where I probably wouldn't want it retrying over again (e.g., bad url). Do you have some specific urllib exceptions that you see?
DeleteI've gotten all kinds of responses over time, including some that made no logical sense. So I check response.status... if it's anything but 200, I go to sleep for a second, close the connection, and loop back to re-establish the connection. And I let the loop run five times before giving up. It has made the process more "robust" for me, for lack of a better word.
DeleteI've posted a new beta version @ https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip .
ReplyDeleteA summary of changes:
- Changed clientUID to be specific to the user and bank url. It's "automatic", with no special user config. Note that the first time you connect, a new clientUID will be created, but then reused indefinitely, unless reset by the user via Setup (see below).
- Modified Setup.py to allow "resetting" an account connection (clears clientUID for user@site). Deleting an account entry also resets the connection.
- Added skipZeroTransactions: Yes/No option to sites.dat. If Yes, $0.00 transactions are removed from statements. Default=No
- Added REINVEST transaction types to _scrubINVsign()
Feel like I'm forgetting something, but I think that's the major edits for now, plus a few bug fixes. Feedback appreciated.
Hi Robert,
DeleteI just wanted to confirm with you that the 03-12-2017 beta version of the scripts work as expected with:
Chase Visa
Discovercard
Fidelity
Regions Bank
The 'Test' using Setup.py for Chase *did* initially throw an error until I verified my identity @ Chase.
The $0.00 amount scrubber seems to also work as expected with Discovercard.
I tried the new 'Delete or Reset' option in Setup, up to the point of where I could (C) Cancel.
I'd say that you've hit another home run. Well done!
I can't thank you enough for your work on the scripts.
-Kevin N.
Hi Robert,
DeleteI meant to add that there is no longer any need for 'His' and 'Hers' < site > entries in sites.dat with regards to having more than 1 account @ Chase.
Again, well done!
-Kevin N.
Worked with all but my citi accounts but I think citi discontinued their OFX recently as I couldn't get it lately with old version.
ReplyDeleteWorked with: Chase, Discover, CapOne360, Vanguard, AmexStarwood
A note on Chase:: I had to re-authorize it after first run...get email, go to secure message and follow instructions.
If anyone can confirm...is Citi OFX dead..might be time to kill a credit card or two.
My bad. Citi is working fine. Not sure what was going on but currently on the beta it is working great. Thank you for all your work...now if only MS would open source MSmoney.
DeleteGreat job Robert!!
ReplyDeleteWorked fine for all my accounts. Since you're looking for feedback thinking you forgot something, the only very minor things I noticed:
1) last line of Revisions in sites.dat:
# -Added skipZeroTransactions
2) your code still addends generic clientUID to sites.dat which may very well be intentional
Thank you so much for bringing up-to-date.
Works fine here, Robert. Including AT&T (run by Citicard and stopped working with last full release version a few weeks ago, Chase (required an online verification), Fidelity, and others for which I have not seen reports of issues here. Thank you for continuing to support this invaluable add-in.
ReplyDeleteFirst, thank you so much for continuing to support PocketSense! It's awesome being able to still use Microsoft Money after all these years.
ReplyDeleteI just upgraded to the new beta after my Citi credit card transactions stopped downloading with the older versions of PocketSense. That fixed the problem and everything is working great.
However, I did find one small bug in the beta. I tried removing an account and I deleted it from the sites.dat file before removing it from PocketSense using setup.py. When I went to remove it from PocketSense, it would crash and close without removing it. To get it to remove successfully I had to put the site back in my sites.dat file and remove it in PocketSense first, before deleting it from the sites.data file, again.
sites.dat holds generic sites data, not account data. Your account data refers to the sites data, hence the crash.
ReplyDeleteThanks for the feedback guys. A couple of notes:
ReplyDelete- I left the clientUID in sites.dat in case someone reverted to an earlier version... and it isn't used by the new version, so no harm, no foul.
- I just now modified Setup to only delete the connection key if the site still exists in sites.dat. That corrects the crash "bug", but... if a person deletes the site before clearing the account, there will be dangling connection keys in the connections file (connect.key), since it can't delete the key without knowing the url. Not recommended ;)
I need to upgrade for Citibank, not sure I want to.
ReplyDeleteDoes this mean We can no longer control what the ClientUID is and have to accept whatever is generated? and how do we get it's value. I need the value for other software so I don't have to go through the new clientUID routine each time.
Raymond
I'll add an option in Setup when listing accounts to include the clientUID in the list. Once created, it won't change. They are stored in connect.key. The username and url are "hashed", but the clientUID is not.
DeleteThe beta originally seemed to work fine for me. I had to re-register both Chase and Capital One, but it was easy enough, and the new code allowed me to get rid of my two PocketSense directories. Citibank and Discover both initially seemed to work well. However, oddly enough, when I ran the full download, my Discover Bank entries, which I had entered manually earlier in the month, did not make me go through the normal Money sequence of matching downloaded transactions to my manually entered transactions... it also showed in the account register as unmatched. However, oddly, if I ran Setup and Test Account, it worked fine, even if I did more than one in the same run of Setup. (I left myself two more Discover Bank entries to test a lter version of Pocketsense.)
ReplyDeleteI have no idea how to code in Python nor do I understand OFX "stuff". However, I retrofit my old changes to your beta ofx.py DoQuery routine to retry five times in case of an invalid HTTP response code. I noticed that the old code used to have a statement that said "if h: h.close()" and the new code doesn't have that... I have no idea what it was supposed to do, nor if it was simply unnecessary, but in light of the post above, I thought I'd just point that out.
DeleteHarold, my Discover transactions are all matching up fine. Perhaps you downloaded some from the web? Those didn't match up for me after the beta trial, I think because the scrubber routine makes some changes to the FID (or datestamp?). I had to manually match those that I had downloaded from the web, but all is well again now.
ReplyDeleteI entered them manually. And it seems really weird to me that it works and matches up when I am running from Setup, but it seems to ignore the downloaded transactions when I run Getdata. It certainly seems like it might be the scrubber stuff, but why should that make a difference?
DeleteIt is odd, since the process is identical... *unless* you enabled CombineOFX, and are packaging all statements into one. If so, I'm guessing that Money is somehow interpreting it differently. I use Discover also, but never entered anything manually during the "outage", and I do use CombineOFX. No issues since resuming downloads.
DeleteSo keep in mind that I don't understand this stuff except generally. I tried adding the h.close statement, but that didn't eliminate the problem. Then I tried deleting the transaction, and adding it back to Money manually, and I got the same behavior... it didn't work with Getdata but it did work with Setup.
DeleteSo then I looked at the downloaded files, and lo and behold the one from Getdata did not have the transaction and the one from Setup did.
The obvious clue that I can think of is that the DTSTART and DTEND values for the file that came as a result of Setup were 2/11/2017 and 3/1/2017, while the values for the file that came as a result of Getdata were 2/28/2017 and 3/02/2017. The problem transaction was a end-of-month transaction with a posting date of 2/28/2017.
I have no idea how or why these dates were set.
I am NOT running CombineOFX. I am running with defaultinterval defaulting to 14, with no overrides. The transactions are for Discover Bank.
Harold
I think I "got it" almost as soon as I hit enter on my prior post. Today is March 15th, so with defaultinterval=14, I should NOT get the transaction from 2/28. I suspect that Setup uses a different logic to specify the start date for transactions and that's why I got the transaction when using the Setup test function.
DeleteH
I believe you do. I misunderstood the original issue, thinking you were getting *duplicate* downloads w/ Getdata, that weren't matching with originals. The download interval for testing in Setup is 31 days, w/ the hope of catching at least one transaction over the previous month.
DeleteSome minor bug fixes I noticed, and added the option to show clientUID (connection key) when listing accounts in Setup. I think it's about ready to put out as "final", but that doesn't mean other ideas won't be considered for adding (e.g., retries, etc.).
ReplyDeletehttps://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip
Not that you even remotely need my code for the retries, but here's my version of DoQuery with the before-newest beta:
Delete# This routine has been changed to retry in case
# an error response is received for the query.
RETRY_INTERVAL = 1 # Retry every n seconds
RETRY_TIMEOUT = 5 # Max number of seconds to retry
DOWNLOAD_SUCCESS = 0
RetryTime = 0 # seconds already retried
# urllib doesn't honor user Content-type, use urllib2
response=False
while RetryTime < RETRY_TIMEOUT:
try:
errmsg= "** An ERROR occurred attempting HTTPS connection to"
h = httplib.HTTPSConnection(self.urlHost, timeout=5)
if Debug: h.set_debuglevel(1)
errmsg= "** An ERROR occurred sending POST request to"
h.putrequest('POST', self.urlSelector, skip_host=1, skip_accept_encoding=1)
h.putheader('Content-Type', 'application/x-ofx')
h.putheader('Host', self.urlHost)
h.putheader('Content-Length', str(len(query)))
h.putheader('Connection', 'Keep-Alive')
p = h.endheaders(query)
errmsg= "** An ERROR occurred retrieving POST response from"
#allow up to 30 secs for the server response (if it takes longer, something's wrong)
h.sock.settimeout(30)
response = h.getresponse()
if response.status == 200: # Good response code
RetryTime = RETRY_TIMEOUT + 1 # Terminate loop
respDat = response.read()
#if this is a OFX 2.x response, replace the header w/ OFX 1.x
if self.ofxver[0] == '2':
respDat = re.sub(r'<\?.*\?>', '', respDat) #remove xml header lines like
respDat = OfxSGMLHeader() + respDat.lstrip()
f = file(name,"w")
f.write(respDat)
f.close()
DOWNLOAD_SUCCESS = 1 # Set flag
else:
RetryTime += RETRY_INTERVAL
print " +Error response from server:", response.status, " Retry time:", RetryTime
time.sleep(RETRY_INTERVAL)
h.close()
except Exception as e:
RetryTime = RETRY_TIMEOUT + 1 # Terminate loop
self.status = False
print errmsg, self.urlHost
print " Exception type :", type(e)
print " Exception val :", e
if response:
print " HTTPS ResponseCode :", response.status
print " HTTPS ResponseReason:", response.reason
if DOWNLOAD_SUCCESS == 0: print " +Error downloading file"
OK... OK... so did I know that this posting board would get rid of all indentation, which is relevant in Python? I don't think there is a way for me to delete a message, or I would delete my sample code.
DeleteH
Robert,
DeleteMany thanks for keeping this invaluable add-in functional. I ran into a problem with my AT&T card (a CitiCard brand) which was fixed by this latest beta. My only problem is the few institutions that don't have OFX servers.
I took the plunge and ran through the beta. It all worked fine for Citibank, BankofAmerica,Chase, and Fidelity.
DeleteAs far as Chase is concerned, I was able to edit the connect.key file and did not have to mess with a new ClientUID routine @ Chase.
Still would be interested to know what dot Net function you used to test with.
Raymond
Harold: I took a few minutes to look at adding a retry option, but I can't test, and I'm not sure what conditions you've had where it makes it to the response check (except an actual response). The only issue I've seen (and can reproduce) are timeouts, which throw an exception.
DeleteI hear you, Robert. Even since putting in the beta, I've been having good days lately, with no errors. If you want to test, though, you can just go back to the version of Pocketsense before the beta and try Discover. Of course, the 403 error will turn out to be non-recoverable.
DeleteOTOH, it's really not a big deal. When I get your code, it typically takes me just a few minutes to integrate my changes.
Thanks a lot for listening, and for keeping this great software going!
Harold
Same here Ron. How do we get PayPal do get on board? :-)
ReplyDeleteJust tried the latest beta. Observations:
ReplyDelete- All accounts connected fine
- I did have to re-register with Chase, which interestingly it was not required with the last beta.
- The Discover $0.00 scrubber works great. Since it is a global setting, rather than an account setting, I'll keep an eye out to make sure there are no unexpected impacts to other accounts.
- Cannot test the REINVEST sign issue until there are new REINVEST transactions ready to come down.
- A small handful of transactions in two different accounts are not getting matched. Even after I deleted the old downloaded copy and accepted the new downloaded copy. I will have to wait a day or two to see how widespread the problem is.
Carlos
Weird Chase issue this (Sunday 3/19) morning.... with next to last beta... CHASECREDIT returned {CODE}2000{SEVERITY}ERROR{MESSAGE}ofx.validationException] sporadically (I replaced angle brackets with braces to allow it to post). It would come back this way... then work fine... then come back this way.... unpredictable. It worked fine last night and last week...
ReplyDeleteHarold
Hi Harold,
DeleteSunday mornings is when many financial institution run general maintenance on their servers. Do you think *that* might be the cause of the anomaly?
I just ran the 12-Mar-2017 version of the scripts without incident re Chase.
-Kevin N.
Hi Harold,
DeleteAfter running the scripts, I ran Moneydance, which also uses OFX direct-connect protocols and it threw the same error that you reported.
The fact that it's happening sporadically across more than 1 application would make it appear to be a Chase issue and not an issue with the scripts themselves.
-Kevin N.
It seems to be working now, some hours later. It's reasonable to assume that it was Chase, although it might be useful to know if Pocketsense was actually sending exactly the same thing and getting different results.
DeleteIs there any easy way to capture the data Pocketsense sends to the financial institution?
Harold
The Chase web site was down at the same time that the ofx.validation error was being returned. Safe to say it was a server-side issue
DeleteHarold, add the group of 5 # lines to ofx.py around line 332, and temporarily un-comment the last 4 of them (indentation doesn't copy correctly):
ReplyDeleteSendRequest = True
if Debug:
print query
print
ask = raw_input('DEBUG: Send request to bank server (y/n)?').upper()
if ask=='N': return False, ''
#log rq file to xfr as *.ofxqqf (remember to remove files from xfr folder)
# qqfFileName = ofxFileName + "qqf"
# qqf = open(qqfFileName,'w')
# qqf.write(query)
# qqf.close()
#do the deed
client.doQuery(query, ofxFileName)
if not client.status: return False, ''
Thanks! Next time it happens, I'll try logging out and seeing if my PC is sending anything different!
DeleteH
Something has changed at Chase since Sunday. Pocketsense does not appear to be affected. I have seen errors in other software (The error shows up as NOT IMPLIMENTED.), but this software continues to run fine. I have been using the combined ofx file and have not noticed anything out of the ordinary.
ReplyDeleteRaymond
I used to download my Discover statements through Pocketsense. Then it stopped working in January as everyone here has noticed. Then last week Discover changed my CC account number. When I now try to download my transactions using the latest Pocketsense beta, the OFX file has an error that I'm not authorized to access the account and I need to register my account "through the online account center". I've already confirmed that I'm using the correct new CC, username, and password in Pocketsense.
ReplyDeleteDiscover helpdesk was useless because I'm not using Quicken. I did manage to find a registration page by digging into the discover.com front page, but when I register my card there, it tells me I'm already registered and offers to change my password. Does anyone know where I can register my new Discover CC for OFX access?
Just to verify... you did delete your account entry in Setup and re-add it using the new card number?
DeleteApologies for the slow reply. I had to delete and readd my Discover account in PocketSense. Simply modifying the existing account did not work.
DeleteAny luck with getting TD Bank to work with these changes? It requires the useragent to be set. Not sure if it worked with previous versions because I just got a TD Bank account.
ReplyDeleteThanks
By useragent, are we talking about the http user-agent header field?
DeleteYup. That's what I read elsewhere on this blog and in the moneysense forum.
DeleteHmmm. I tested a the sites I use with and w/o a user-agent defined. All but Discover work with or without. The decision, then, is whether this should be a parameter that defaults to off or on. Since Discover is probably one of the most widely used, and most servers don't seem to care, I'm leaning towards a site option to enable the user-agent field, but default to none.
DeleteAnyone have an opinion?
Scratch that. I'm just going to implement a retry w/ the user-agent enabled, if it fails without. It may be that a site requires a specific user-agent name. If so, I'll just add it as a site option.
DeleteWe need to learn more about that field. Sounds like a good strategy Robert!
ReplyDeleteMaybe that is a needed field for other sites that don't work with Pocketsense, like Merrill Lynch?
Any help you can give with getting Merrill Lynch to work is appreciated. Pardon my ignorance but what exactly is the user-agent field? Is it something specific and unique to each financial institution? If so, where might they be listed?
ReplyDeleteThanks.
I didn't know either, but apparently it is part of the request header and identifies the application requesting the information from the server, typically a browser, and helps the hosting server determine the best way to provide the requested information. Robert may have to use a dummy browser identifier??, since a Python script is actually making the request using the appID=QWIN, identifying the requesting App as Quicken.
ReplyDeleteBeta posted for testing. Implements a retry w/ user-agent if download fails without it.
ReplyDeletehttps://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip
As for the user-agent header, it's a standard http field, and can be used to identify the client (e.g., Chrome, etc.). The beta simply uses "PocketSense". If that doens't work, and if someone has a specific value that's needed, then I'll need to revisit.
Your willingness to address these rare issues is really appreciated. And I can't believe how quickly you put out a beta. Unfortunately, I tried it with Merrill without success. Here's the OFX file it returned:
ReplyDeleteOFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:0dca0c1c-f736-4b25-b985-7ffccd3e14d8
:OFX::SIGNONMSGSRSV1::SONRS::STATUS::CODE:2000:SEVERITY:ERROR:MESSAGE:Your request cannot be completed at this time, for assistance please contact us at 1.800.MERRILL (637.7455).:/STATUS::DTSERVER:20170410183148.913[-4:EDT]:LANGUAGE:ENG:FI::ORG:Merrill Lynch & Co. Inc.:FID:5550:/FI::/SONRS::/SIGNONMSGSRSV1::/OFX:
(In order to post the OFX file here I had to replace the left and right carat marks with : as the carats were being interpreted as HTML code).
I'll be interested to see if anyone else has better luck.
Have you tried ofxVer=102 ?
DeleteJust tried ofxVer=102. Same exact error message except it shows Version:102. Oh well :(
DeleteCurious about the next part, but have no way to test. Change line 296 in rLib.py to the following (ver 1.x header definition):
DeleteSECURITY:TYPE1
I edited rLib.py as indicated, ran it, checked to be sure rLib.pyc showed an updated time stamp. Then tested the Merrill account and, interestingly, the OFX file still showed SECURITY:NONE. I'm pretty sure I did everything correctly. I'm guessing Merrill isn't looking at that setting and just returns a canned response.
DeleteSince I don't use Merril, I can't really test. Another quick check would be to see what response you get using OFX 2.x. I started adding support when debugging the Discover issue. Set ofxV=211. I doubt this is the problem, but easy to check.
DeleteRobert, thanks for your efforts. I changed ofxver from 102 to 211 in my sites.dat file. The test didn't generate an error, but when I imported the file in Money I got the message: The file you attempted to import appears to be invalid or contains corrupt data....... I took a look and all I got was the header info (see below); there was nothing else in the OFX file.
DeleteOFXHEADER:100
DATA:OFXSGML
VERSION:102 (I know it says 102 but it's 211 in sites.dat for Merrill)
SECURITY:TYPE1
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
I'm going to stick with just downloading the OFX file directly from the Merrill site. I don't have many transactions so don't need to do it very often. Thanks again for your help.
After posting I realized you may have meant to change Version in rLib.py so I changed that. Same issue as above, but header now says VERSION:211.
DeleteWow. That's amazing how fast you got that up. Thanks a lot! And that did fix TD Bank!!! Much appreciated
ReplyDeleteAnd just to confirm, it works great with all my other accounts also!
ReplyDeleteHi Robert, Thank you very much for your great work! The add-on works great on discover and citi credit cards.
ReplyDeleteThere is an issue with chase bank account. It shows "The file you attempted to import appears to be invalid or contains corrupt data" in Money. I have enable the access in chase bank. Is there something wrong with the site.dat or something missed?
The following is the site.dat (from moneydance)
SiteName : Chase
AcctType : BASTMT #bank
fiorg : ISC
fid : 1601
url : https://www.oasis.cfree.com/fip/prod/01601.ofx
bankid : 123456789
brokerid :
appid : QWIN
appver : 2400
mininterval:
timeOffset :
Another question is where can I find Sam's Clue credit card's site data?
Thank a lot!
Hi All,
ReplyDeleteScripts Ver: 10-Apr-2017 "Beta"
Is any else seeing this odd behavior with Discovercard?
Recently, I have been unable to d/l from Discovercard using the scripts. The error thrown was:
'Access Denied. You don't have permission to access "http..." on this server.'
I use the scripts with MS Money. I also use Moneydance. (I'm able to d/l Discovercard with Moneydance)
Looking at the Discovercard account in Moneydance, where '#' represents each digit of the card number, the credit card number syntax has changed from:
'#### #### #### ####' (which represents the card number)
to
'Discovr##X##X####X####'
That is not a typo. The beginning sequence is 'Discovr' (without the 'e') - the X's are just X's. Again, the '#' represent numbers.
What is particularly odd is that some of the numbers in the new syntax coincide with the card numbers and some do not and seem completely random.
e.g. The first 2 digits coincide with the 7th and 8th digits of the credit card.
The second set of 2 digits does not coincide with any 2 digits of the credit card.
The third set of 4 digits does not coincide with any 4 digits on the card.
The last set of 4 digits coincides with the last 4 digits on the card.
I changed the account number in the scripts to the new syntax and numbers and I was then able to d/l from Discovercard.
-Kevin N.
Thanks Kevin. I noticed the same, and posted in the latest post/comments just before you ;)
DeleteHi Robert,
DeleteThank you for your reply and confirmation.
-Kevin N.
I am getting Access Denied on Discover too after downloading the May 11 update. I use MS Money and PocketSense. I tried changing the syntax to what was posted above with the XXs and random numbers, but no luck.
ReplyDeleteWhat else can I try? Been at this for 3 hours now, so happy to have any suggestions. Net
Either go to the newer comments, or read my blog article at: https://microsoftmoneyoffline.wordpress.com/2017/05/12/discover-has-changed-your-account-id/
ReplyDeleteRobert,
ReplyDeleteI've been using Pocketsense for several years. Love it! Yesterday I downloaded and installed your latest version (11-May-2017 18:50 EST). I can download stock and mutual fund prices but I'm having problems downloading OFX data from my credit cards and Fidelity accounts. I'm seeing the mesaage below. Any suggestions would be much appreciated.
** An ERROR occurred sending POST request to online.americanexpress.com
Exception type :
Exception val : endheaders() takes exactly 1 argument (2 given)
The ofx post header was modified for Discover, with the assumption that other banks wouldn't mind. Looks like you've disproved that assumption ;) I'll modify to support both header types.
DeleteRobert, thanks for the quick response and your continued support of Pocketsense.
DeleteTry and let me know.
Deletehttps://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip
Robert, I downloaded the beta zip and installed it like a typical, new version of PocketSense. I then used the Setup.py to test OFX retrieval from American Express. It failed with the following error message.
Delete** An ERROR occurred sending POST request to online.americanexpress.com
Exception type :
Exception val : endheaders() takes exactly 1 argument (2 given)
An online error occurred while testing the new account.
Could this be an error in my execution?
Thanks
-jim
I don't use AMEX, but configured a dummy account and tested using settings on ofxhome. It connected fine (i.e., made it through the failure point you're hitting without issue). It's either something odd about the account/username info, or your Python version.
DeleteHighly recommend updating to latest version of 2.7.x Python. I've used them all, but great success w/ the Anaconda distro.
Robert, I upgraded to Python 2.7.13 and that solved the problem. I reinstalled your your latest version (11-May-2017 18:50 EST) and everything is working again. Thanks for your quick responses and continued support of PocketSense.
DeleteNxtTek,
ReplyDeleteI went to the blog you posted and followed that. It works now! Thank you! :)
Pocketsense has been working for me for several years now and I really appreciate all your work.
ReplyDeleteYesterday I hit a problem connecting with my Wells Fargo brokerage account. The request to connect is rejected with a message saying "sslv3 alert handshake failure". This problem has continued into today. Have Wells Fargo made changes to something?
Thank You
Please ignore. The problem has now disappeared!
DeleteHi,
ReplyDeleteYou guys helped me with my Discover issue, so I am hoping that you can help me with Chase.I use pocketsense with MS Money Plus. I get the following error info on my xfr file:
/OFX//SIGNONMSGSRSV1//SONRS//STATUS//CODE/15510/SEVERITY/ERROR/MESSAGE/Please verify your identity within the next 7 days. Using your desktop computer, go to your bank’s website and visit the Secure Message Center for instructions.//STATUS//DTSERVER/20170528175341.231[-4:EDT]/LANGUAGE/ENG/FI//ORG/B1/FID/10898//FI///SONRS///SIGNONMSGSRSV1//CREDITCARDMSGSRSV1//CCSTMTTRNRS//TRNUID/2ddb49e0-6769-4c16-b1c8-58898052ddf0/STATUS//CODE/15500/SEVERITY/ERROR//STATUS//CLTCOOKIE/4//CCSTMTTRNRS///CREDITCARDMSGSRSV1///OFX/
Not sure what all that means, but I tried to go to my website which lets me login just fine, and look for a message in secure message, there isn't one. I do have two chase accounts to set up, myself and my husband, but I can't even get one to work anymore.
I am using this setup on my sites data:
SiteName : CHASE
AcctType : CCSTMT
fiorg : B1
fid : 10898
url : https://ofx.chase.com
ofxver : 103
bankid :
brokerid :
appid :QWIN
appver :2400
mininterval
Any ideas, thoughts or pointers in the right direction would be appreciate.
Net
Hi Net,
DeleteDid the connection to Chase via the scripts work as expected in the past? OR Are you setting it up for the first time?
Have you enabled your account(s) for 'Linked apps and websites' at Chase.com?
Have you received an email from Chase which also states that you need to verify your identity?
You stated that you have two accounts @ Chase; have you tried logging on to Chase.com for the other account to see if the secure message center there has the reported message?
-Kevin N.
Kevin,
DeleteThank you for the quick reply. Walking through those steps was helpful. Chase was downloading just fine until May and I do have the financial software enabled on the Chase website. I had not thought to look at email, but today I checked and there was a message there. However, I did a search and had no previous emails from Chase. I checked secure messages and found the message there for the first time. It too was the first time one was listed. I do not know what changed and why they didn't appear before. But both accounts are now working fine now. Thank you!
Only account now that I do not have downloading is Barclaycard. Do you know if there is any info posted anywhere about that bank? I have to download it manually. I do not see it in the OFX directory.
Hi Net,
DeleteI'm glad to hear that you were able to get your Chase accounts working.
Unfortunately, I have no knowledge to share re Barclaycard.
-Kevin N.
Hi Robert,
ReplyDeleteI'm getting duplicated download transactions from Discover Card and Bank, which aren't matching with previously-downloaded items. Normally subsequent downloads don't exhibit this issue (even with overlapped date range). Thoughts?
Hi Derek. I wasn't having the issue a few days ago, but I'm not in a place where I can test at the moment. I'll check again in a few days, but I'm wondering what version of Python you're running? If it's not the latest 2.7.13, I'd recommend updating, as there seems to have been some issues w/ older versions.
DeleteWas on 2.7.2.5 but just updated to 2.7.13, and the issue still occurs. (And from what I can tell, version 3.x isn't advisable?)
DeleteFinally able to test, and don't see that issue. Do you see the following message during download? I guess I should also ask if you're using the Sunset edition of Money, or an earlier one. I only test on the Sunset version.
Delete+Scrubber: Processing Discover Card statement.
Re Python versions, you're correct that the scripts are written for version 2.
Using Sunset, and yes Scrubber statement appears for Bank and Card.
DeleteUnfortunately, I can't replicate the problem. I'm assuming that we're only talking about transactions downloaded via the scripts, and that *none* have been manually downloaded/imported from Discover? The transaction ID values assigned by the script downloads will never match what Discover provides natively via their manual (web site) download option.
DeleteI thought I only downloaded via scripts but realized I may have manually downloaded/imported from Discover when obtaining their new acct # format. If that was the case, perhaps all subsequent downloads via scripts would cause the dups, but future transactions should no longer exhibit this?
DeleteDerek - all downloads via scripts should work properly, but as Robert stated, those transactions you downloaded from the website will repeat when the script obtains those. Probably best to delete the web downloaded transactions as you find the repeats OR shorten the date range.
Delete-ameridan
About 10 days ago I began to have problems with the E*TRADE download with prices in Money showing up as 0.
ReplyDeleteIn the section, prices are 0, with 0
Anyone else seeing the same issue?
Regards,
Michael
Blog post eliminated bracketed section names
ReplyDeleteIn the INVPOSLIST section, UNITPRICE is 0
This must have been an E*TRADE issue, the issue has been fixed and prices are now downloading
DeleteThanks for posting the result.
DeleteApparently, there is a Wells Fargo OFX connection issue in progress. Pocketsense has not been able to connect since the weekend. Here is more from the Moneydance forum:
ReplyDeletehttp://help.infinitekind.com/discussions/online-banking
Apparently, Wells Fargo might have made some changes. No updates yet...
Hi David,
DeleteTake a look at this post from OFXHome forum member humblepie:
http://www.ofxhome.com/ofxforum/viewtopic.php?pid=108477#p108477
Essentially, he says to change the appver from 2400 to 2600 and change ofxver to 103 in sites.dat.
I'm not a WF client so I don't know if it'll work or not.
Post back with your results. OK?
-Kevin N.
Thanks Kevin. No, this did not resolve the error. BTW, this is the message:
DeleteWELLS FARGO : xxxxxxxxxxx : Getting records since: 20170522
** An ERROR occurred sending POST request to ofxdc.wellsfargo.com
Exception type :
Exception val : [Errno 1] _ssl.c:499: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
Hi David,
DeleteSorry to hear that it was of no help.
The error msg. may prove helpful to Robert.
From what I've been reading on this matter, the general consensus is that WF now requires two-factor login and *that* may be causing the sudden errors.
-Kevin N.
Agreed, I think this is more than a config issue.
DeleteOk, I resolved the issue. I did some digging around with the ssl error. So, Wells Fargo must have upgraded the SSL services on the ofx server. So, I found this article:
Deletehttps://stackoverflow.com/questions/35405092/sslerror-sslv3-alert-handshake-failure
Sure enough, I was running Python 2.7.8, and only Python 2.7.9 and later has the SSL feature updates (SNI is the feature). So, I would just load the latest 2.7.x Python if you see this condition. And, you have to add the following to the site.date for WF Entry:
ofxVer : 103
Thanks!
Hi David,
DeleteAh, OK. I'm glad to hear that you were able to get it working.
I forget what problem it was that I was having but it also came down to using an older version of python.
I'm at v. 2.7.13.
-Kevin N.
Good Motning,
ReplyDeletePocketsense has been working for me for several years now and I really appreciate all your work.
This morning I have hit problems with all my Schwab Brokerage accounts which caused my regular we login to get suspended. The pertinent messages in the OFX file shows:-
"CODE" 15500
"SEVERITY"ERROR
"MESSAGE"Signon (for example, user ID or password) invalid (ERROR)
I got my password reset and was able to access the accounts through the web-site, but when I changed the password in Pocketsense and retried the scripts the problem was still there.
I'm not sure if Schwab has made changes to their OFX interface or whether it's due to the holiday? Anyone else hitting this issue?
Thanks & Happy 4th!
Thank You
I would have just waited until tomorrow. Sundays and Holidays are when they do server maintenance.
ReplyDelete