Robert, thanks for starting a new thread for comments; the "load more" was definitely causing a problem.
For those who discover the issue with downloading for AMEX credit cards, please refer back to the previous thread which can be found at the posts beginning with packpike on August 8, 2019 at 10:15 AM near the bottom (after "load more") at https://pocketsense.blogspot.com/2018/08/new-version-available.html
There are also loads of other topics addressed in that thread so you can search there before posting your issue :)
The "load more" does not work for me. The last comment I see is from Joe@TheWho.orgMay 29, 2019 at 7:37 AM.
Any way to get to those comments after May 29? Any way to post the AMEX card fix here? I think I may have mistakenly asked multiple times because I couldn't load more comments.
Load more should be an active link underneath the comment box at the bottom of https://pocketsense.blogspot.com/2018/08/new-version-available.html I just checked it and it still works. If it's not active for you then it's something on your system.
Here's what I previously posted:
Ken August 8, 2019 at 10:39 AM You're not alone. I tried to update my Amex accounts and ran into the same problem. Figured it was a temporary glitch so didn't pay much attention. Thanks for taking the next step and finding out it's a system change on Amex's side. The solution for us is actually quite simple. Just delete the account from PocketSense and then add it back in. You'll notice that when you enter your login ID and password to add the account, the list of your Amex accounts will look quite different than they used to. Similar to Discover, they're now using non-account numbers.
Make sure you're using the most recent PocketSense or it won't work.
I think there's something wrong with my system (preventing that javascript code from running to load more). I was able to use another computer to see the rest of the comments.
Microsoft Money Sunset Edition still works fine on my Windows computer. I've been using Microsoft Money for many years and it's like a comfortable old shoe. However, I've gotten worried lately. I've invested many thousand of hours of my life tracking my personal financed in Microsoft Money, I have transaction data in Microsoft Money all the way back to 1992. It all sits in a proprietary data format accessible only with software that was abandoned by Microsoft years ago.
I check out all the recommendations I could find for MS Money replacements, and they all had proprietary data formats. Then I found Tiller ( tillerhq.com ) Tiller uses a Google Sheets spreadsheet for transaction data and does really slick and automatic updates from my financial institutions. Since the data is all in a spreadsheet format, it is "future-proof" and not in danger of loss like proprietary Microsoft Money data. I've been using Tiller for a month now, and the more I use it, the better I like it.
The process of getting my financial life going back to 1992 moved over to Tiller-compatible spreadsheets was non-trivial. I probably spent over 8 hours doing this. I had dozens of old credit card accounts, 9 checking accounts, and many investment accounts in Microsoft Money that all had to be exported and converted, one by one. As an aid to myself during this process, I wrote a cheat-sheet. Today I updated my cheet-sheet into a full step-by-step instruction for this process. It resides in a Google Sheets spreadsheet and here's the link https://docs.google.com/spreadsheets/d/11N5Iz_Hqz-JABeSVTXF2SttXHyfnvZXcACAl81s_7vs/edit?usp=sharing
This alternative to migrating away from Microsoft Money has the advantage of being accessible to fairly non-technical people. I just thought you might be interested.
Anyone having a problem with Amex downloads - running into this for the past 2 weeks - no changes in system. The output I get is (note I had to replace CODE with "----" to allow this post): <---->0INFOLogin successful20190831075541.301[-7:MST]ENGAMEX3101FMPWebBCM201908310755415ceedf48-f311-4a4d-9c44-297403c2a1d1<---->2003ERROR4
I manually download and import QFX files from TD Bank credit card. They updated their website a few weeks ago and since then Money can't import the file.
Turns out the file now includes pending transactions with an empty FITID, and transactions with empty SIC fields that Money can't process. Also all debit transactions have positive amounts, so Money interprets them as credits.
I added the following regexes in combineOfx to fix these issues:
``` # remove all pending transactions with empty FITID c = re.sub('(.*?<.*?)', '', c)
# remove empty SIC lines c = re.sub('<', '<', c)
# make all debit transactions negative so Money doesn't treat them as credits c = re.sub('(DEBIT.*?)(\d)', '\g<1>-\g<2>', c) ```
I use something similar to the following script to process and import manually downloaded files:
``` import os from control2 import * from rlib1 import *
Roman: Your code to remove transactions meeting a criterion is a whole lot simpler than mine. I suspect two things: 1.When you post, the forum removes things between < .... > and big pieces of your expressions get removed. If you replace all < with < and > with > then that problem will not happen. Some replace those characters with curly braces {} for posting purposes.
2. Simple expressions often match a lot more than you think and can remove more than one transaction. Watch for that to occur.
1. Ah, you are right, didn't notice it last night. Too used to the convenience of triple backticks in Slack and such. Reposting with brackets properly escaped.
2. I agree that regex can be quite dangerous. I used greedy matching, but it sure can break if transaction tags are not closed properly or if fields in a different order. The proper way is to split the file into transactions and then check each transaction separately, but I was just happy to get it to work. As the saying goes: "Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems."
Regexes in combineOfx to fix TD Bank OFX: ------ # remove all pending transactions with empty FITID c = re.sub('(<STMTTRN>.*?<FITID><.*?</STMTTRN>)', '', c)
# remove empty SIC lines c = re.sub('<SIC><', '<', c)
# make all debit transactions negative so Money doesn't treat them as credits c = re.sub('(<TRNTYPE>DEBIT.*?<TRNAMT>)(\d)', '\g<1>-\g<2>', c) ------
Process and import manually downloaded files: ------ import os from control2 import * from rlib1 import *
My CitiCard account is the first time I had to go to OFXver 103, which I think involves the use of ClientUID. Then with the Citicard website, you click to a place that starts a 10 minute window where a new desktop client can get in.
I am thinking the ClientUID gets remembered during the 10-minute window, and then you don't need to use the web access subsequently. I think the ClientUID is a bit secret -- should not be posted. Other things in the sites.dat file are not at all sensitive, and before ClientUID, that file could be freely shared. I am thinking that the ClientUID would have better been put into the always-sensitive/secret ofx_config.cfg file. Of course the ClientUID is just one factor in a 2-factor authentication method. Just a thought. I am very happy that the CitiCard transactions download without drama now, and I can remember to consider my sites.dat sensitive.
Is anyone else having issues with Discover Card? For the past few days, I'm receiving an 'Invalid OFX statement received' error. The contents of the OFX file are as follows... (replaced < and > with [ and ])
I'm having the same problem so reset the connection. That didn't help so I deleted and tried to re-add the account. Here's what I got: An error occurred requesting accounts from the site. Please check username and password.
I'm guessing either their OFX server is down or they changed the URL. Will try and re-add the account in a couple days. If either of you start getting downloads again, please post a message here.
They are indeed having problems, although I don't understand their terminology, so I can't figure it out. https://community.quicken.com/discussion/7858430/discover-card-and-discover-bank-direct-connection-error-ol-294-a
Having same issues with Discover, although I started having them a bit later than those in the Quicken thread. Quicken and Discover pointing fingers at each other. Some people in the Quicken thread indicated that they had to update their "Financial Institution Branding and Profile". Sounds like they are either changing servers or login methods, but not clear how yet.
Per a post on the Moneydance forums, Discover is no longer supporting non-Intuit clients. https://infinitekind.tenderapp.com/discussions/online-banking/14201-unable-to-download-discovercard-transactions/page/1#comment_47655016
It seems that the latest preview version of Moneydance MD 2019.4 (1892) is able to connect to Discover Card. https://infinitekind.tenderapp.com/discussions/online-banking/14201-unable-to-download-discovercard-transactions#comment_47804342
Discover update. Still doesn't work in Pocketsense. I tried setting up in my test Quicken application. Set it up as Direct Connect, not Web Connect. I look a the logs, and there is definitely interaction with ofx.discovercard.com. First attempt with Quicken didn't work, but second downloaded just fine.
Some more information for those more techie minded. I did some packet tracing. When PocketSense goes out it queries ofx.discovercard.com directly, which is really an akamaiedge.net cloud server.
When quicken claims to query ofx.discovercard.com (according to its own OFX logs) it first talks to services.quicken.com and then to a completely different akamaiedge.net server.
Again this is using Quicken's direct connect. So one thing is clear, Quicken is doing something different. I don't know if it has dedicated servers on the Discover, or if it is using its own proxy like system or what. I haven't tried something else like Moneydance.
Carlos: Could send a full trace log of the Quicken connect, sanitized of personal info to pocketsense1 at gmail dot com? It may be that the first interaction was key also, so if it's possible to remove the account in Quicken and add again and get logs for both connections, then that may be better.
Carlos, I think all Quicken queries first go to their servers, so that they can verify a current paid subscription for INTU.BID 9625; INTU.USERID xxxx. You may be correct though, in that an entirely new url may be in use for Discover, that is unknown to us.
Hm, I still have an active Quicken subscription for troubleshooting. In the options for "connection setting", I see "Express Web Connect" and "Web Connect"; I don't see an option for "Direct Connect". I wonder if there's a way to set "Direct Connect" that I don't see. If I can get Quicken to work, I'll happily send over some HTTP(s) connection headers.
Redmondman, https://community.quicken.com/discussion/comment/20029957#Comment_20029957 comments by digitalmediaphile may get you to Direct Connect for some log-worthy action. I don't follow the steps, myself.
Fiddler tracing adds to what Carlos has mentioned. Quicken does not connect directly to ofx.discovercard.com, but instead via a proxy. This hints at some serious, OAuth-type authentication that may be difficult to duplicate. The rest of the headers look standard. You'll recognize 7101 as the FID for Discover, of course.
POST https://services.quicken.com/ofx-secure-plus/7101 HTTP/1.1 Authorization: Bearer [a nice long auth token]
Just to corroborate Carlos, for Citibank (by contrast, for example), CONNLOG.TXT shows that Quicken is talking directly to 'https://mobilesoa.citi.com/CitiOFXInterface'.
Vs. Discover, where CONNLOG.TXT shows that Quicken is talking to 'https://services.quicken.com/ofx-secure-plus/7101' and never directly to 'https://ofx.discovercard.com'.
I'm curious whether anyone knows if other financial software besides Quicken is working with direct Discover downloads? I only use PocketSense with MS Money so don't know if we're the only ones still experiencing issues. Does anyone follow support issues for other apps?
@ken, I use Moneydance and it can successfully download Discover transactions. On a semi-related note, many people are having problems using Moneydance to download Citicard transactions, which is why I'm testing Pocketsense.
Larry, I was able to get Citicard working with Pocketsense. Follow https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/comment-page-2/#comment-11241 to see how. It's Discover that has stopped working for us after all these years.
Can you confirm that Moneydance is indeed connecting to Discover Card?
I'm using MD 2019.4(1886) which currently is NOT connecting.
When attempting to connect the DC, MD does not display the usual visible error message window. Instead you need to view the contents of the console window to see that the connection failed.
uh oh, we've gotten some OFX with no headers: [HTML] Unhandled OFX message set: HEAD Unhandled OFX message set: BODY uh oh, we've gotten some OFX with no headers: [HTML] Unhandled OFX message set: HEAD Unhandled OFX message set: BODY account: Discover; serviceType: 5; service: Discover Card; mode=both
My Bank of America (BofA) accounts stopped working, both checking/savings and credit cards. The ofx file says "Invalid username/password combination has been entered. Please try again or sign-in to bankofamerica.com to restore your connection from your computer." I updated to latest PocketSense and Python, still same issue. I read the thread from 2017 referencing this issue, but none of the solutions work. My sites.dat entry: SiteName : BANK OF AMERICA AcctType : BASTMT #bank fiorg : HAN fid : 5959 url : https://eftx.bankofamerica.com/eftxweb/access.ofx bankid : 011000138 ofxVer : 103 appid : QWIN appver : 2400
No problem with BoA on my side, although Bank ID is different. I have two different accounts with bankID 052001633 and 051000017. Both working. I leave appid and appver empty
I've had no problem with either BofA checking or credit card account downloads. Here are my sites.dat entries:
SiteName : Bank of America AcctType : CCSTMT #credit card fiorg : HAN fid : 5959 url : https://eftx.bankofamerica.com/eftxweb/access.ofx bankid : brokerid : ofxVer : 103 appid : appver : mininterval :
SiteName : BofA Checking AcctType : BASTMT #bank accounts fiorg : HAN fid : 5959 url : https://eftx.bankofamerica.com/eftxweb/access.ofx bankid : 081904808 ***This should be the ABA routing number off your checks*** brokerid : appid : appver : 2400 ofxver : 103 mininterval :
Be aware that for checking the BankID should be the ABA routing number shown on your checks. If you still can't download then I'd delete the accounts from your account list and re-enter them using just the login ID and password to get the account list from BofA.
import scrubber as s1 #can then invoke s1._scrubFidelity(ofx) for example. #ofx_mod=s1._scrubFidelity(ofx) import os, sys, glob, time, re import win32gui as w32 from win32con import *
ofxfile1= open(sys.argv[1],'r') # open for read in current directory # Input file read in as parameter. Output file fixed in this case. ofxfile2= open("ModOut.ofx",'w') # open for write in current directory
ofxfile1.close() ofxfile2.close() print "Modified version of ", sys.argv[1],' written to "ModOut.ofx" file.' ==============end python file==================
Thanks for your stand-alone version! I have an OFX file from Wealthfront where they use BUYSTOCK for the cash sweep account but it's actually a mutual fund and should be BUYMF (the SELLs and INCOME are appropriately MF entries). I wrote a small REGEX that fixes it but had to manually copy/paste the file into a REGEX parser. Now I'll write a quick script that runs it automatically.
Also having problems with discover card and discover bank ofx downloads for last week or so. However it is even more than that. I cannot log in to the app or website either. I called them up and they opened my account to check it and boom right then on the phone app and website worked again. They said everything looked good... I hung up and player around a bit then tried to download ofx files again and right away app and website stopped working and would not take my login again. I called discover back and they opened up my account again to check it and again boom both app and website started working again, I said "I wonder...", tried to download statements again with pocketsense and boom instantly locked out again from app and web site.
I told them all this on the phone but who knows if this info ever gets to the right people. But not only can I not download my statements it hoses my account so I cannot log in.
Ok I appears to be time based. If I attempt to use my normal script to access my 1 discover credit card and 7 discover bank accounts I am locked out of my app and web login for appx 15 minutes then all is back to normal. Its strange but true. I would guess it is some sort of security trigger I am setting off.
I added some functionality over the weekend to support importing downloaded ofx statements via the getdata.py script. While this isn't needed for some sites, some (like Discover) need the additional "scrub" ;) For anyone interested, a beta/test version has been uploaded for testing.
A few notes about how it works:
1. Setup now creates an ./import subfolder (alongside ./xfr, etc). Running setup creates the folder, but you can add manually too. 2. Any ofx, qfx, etc. file can be dropped into folder. No need to rename. 3. When Getdata.py is run, it will process *all* files in ./import, and if it "looks like" an ofx file, it will be treated the same as a downloaded file (scrubbed, combined, etc). 4. If accepted as a valid ofx file, it is moved to the ./xfr folder (i.e., it is consumed).
An attempt is made to figure out what sites.dat site entry to use by looking for a matching FID or BANKID value in the statement. If either of those are found and match an entry in sites.dat, then the site is selected. This matters when applying site-specific scrub routines.
There are a number of possible issues with importing a downloaded file in this manner, but it should work for most cases. The oddest I've seen is a manual Citi CC download not including the account number, but instead looking something like "xxxxxxxxxxxxxxxx1234".
For DISCOVER credit card users, manually downloaded statements use a different FID value than the direct-connect version. For now, I've deleted my Discover accounts in Setup, and added the following entry to sites.dat to correctly match to a manually downloaded discover CC statement. The Discover *bank* entry is the same for both the ofx and manually-downloaded statements.
Discover changed something on their site that's breaking PocketSense, Quicken and Moneydance. I'm guessing we'll need to wait for reports of success for at least Quicken before Robert (or any of us little helpers) are able to assess PocketSense. In the meantime, this is a handy convenience for those of us working around the issue by downloading the QFX files from the website.
Judging from observation, Discover is now blocking access to ofx.discovercard.com to all entities except for those they have an OAuth trust relationship with. It's a good security measure, and big guys like Quicken will be able to make it work (once they work out the kinks), but if I'm being pessimistic, it may be the end for homebrew scripts like PocketSense.
It may be way worse than just Discover. I'm now getting the same "Access Denied" from Fidelity. If I can't fix this I'm completely screwed as that's where 90% of my transactions take place and there's no way to initiate downloads from their website.
Happening here, also, with Fidelity starting this morning. Error in OFX file:
Access Denied You don't have permission to access "http://ofx.fidelity.com/ftgw/OFX/clients/download" on this server. Reference #18.c600fea5.1568726378.14abd986
I love the development version's ability to import manually downloaded OFX files. Can you add an entry to SETUP for testing that folder (ie Just like we can test each online account and stock quotes we could have a way to run the importer routine)?
Here's my example Wealthfront for sites.dat. I used a localhost address to ensure that we wouldn't try to actually connect to a website. I included wealthfront in the URL to easily add a scrubber hook for siteURL. I pulled the fid and fiorg from my Wealthfront OFX file.
import scrubber feedback. Easy to setup and worked fine using Discover CC ofx download. Thank you again for all of your effort on this program. Was hoping we'd see Discover direct connect start working again, but sounds like the OAuth might kill it for us.
Robert, I haven't tried it out yet, but Google automatically places the downloaded file in Downloads. Couldn't your routine just work from that folder?
It won't work in py3 without some code changes. Usually these are pretty easy, just changing the py2 modules to py3, and fixing up all the print statement to print functions. Out of curiosity, I spent some time yesterday walking through the basics of conversion. I didn't use 2to3 tool, just made hand edits (and regex replace) to fix up print statements and other things to make pylint happy. Still did not work at the end, due to some unicode problems on the pickle load. I was trying to re-use my working config files, and I may have to start a fresh testing environment.
I've started the upgrade a couple of times, starting with 2to3. I think there are issues with the decryption and some of the File IO. I have a version working without encryption, but was able to get the file i/o working yesterday. Just a couple of .encrypt() and .decrypt() additions were needed in ofx.py
Robert, is this something you want help with converting, I could work through it some more if so?
Thank you for the offer. Are you working w/ the latest (dev) code ?
My main concern (and delay) w/ moving to V3 is that it will need to be a clean break, as we wouldn't want two independent code sets, and (probably) don't want to deal w/ having "dual support" in one code base either.
Robert, I hear you on the dual support. It looks like I have a working version from the beta, now for my purposes, and can share after finding a location to post.
For Kevin, to get it to work I had to remove password encryption and delete and then regenerate the connect.key. The pickle.load seems to work when the encrypted data is removed.
files are here for now. https://github.com/bk-test/PocketSense2to3
Robert the new script works great with Discover. Add to your notes, that you should delete the non-working accounts (Discover) in Setup at the same time you are revising sites.dat, so that getdata doesn't hiccup.
Just got off the phone with a Fidelity tech support rep. she said they were aware that quicken users were having a problem and they were working to resolve it. I told her that I used pocketsense and she said although it isn't supported by them, fixing quicken will probably fix pocketsense. Let's hope she's right.
They have test platforms running quicken and she indicated that restarting their PC seemed to fix the issue. She was definitely knowledgeable and very competent.
I won't have access to my computer for a week and hope it'll be resolved when I get back.
Citi worked fine for me. I just did a re-test to confirm. ------------------------ SiteName : CitiCard AcctType : CCSTMT #credit card fiorg : Citigroup fid : 24909 url : https://mobilesoa.citi.com/CitiOFXInterface bankid : brokerid : ofxVer : 103 appid : appver : mininterval: timeOffset :
CITI is still down for me too. I updated to the new url, but no go. I saw on an OFX site that you have to log-in to your CITI account and enable 3rd party access. I did that (it gives you a 10 minute timer) and still couldn't connect via pocketsense. I deleted the account and tried to add it again, but no luck.
CITI is down for me also. (It is an AT&T account but serviced by CITI and had been working OK until yesterday, when I started getting the Access denied messages. Today I received a message, in the OFX download: "We've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement." That link took me to the previously mentioned page where you are allowed 10 minutes to set up the account. But it did not work. I've tried OFX 102, 103 and also the "new" URL: https://mobilesoa.citi.com/CitiOFXInterface. Suggestions appreciated.
1. Use a recent enough version of PocketSense that has the functionality to query the institution for your account list. 2. Execute the account list query within the 10 minute time window.
It's not sufficient to execute an account detail pull, you must execute the account list query within the 10 minute window.
User name : xxxxxxxxxxxxxxxxx Account password: xxxxxxxxxxxxxxxxxxxx An error occurred requesting accounts from the site. Please check username and password.
I used the same name/password I use to sign into the website.
I didn't delete the account in Setup. I just switched the URL in my sites.dat, then started the 10 minute window and all was well again with my Citi card.
After trying various changes, I also eventually got Citi credit card downloads working, so I thought I would describe what worked for me. This is for a Costco Citi Visa card, and I am using the PocketSense August 24, 2018 scripts, so that is as current as it gets.
Here are the sites.dat parameters that are working for me:
Before making any sites.dat changes I was receiving messages in the Citi OFX file that said: "Access Denied" and "You don't have permission to access https://www.accountonline.com/cards/svc/CitiOfxManager.do on this server." That is the url that was in sites.dat, and I was also using the default for ofxVer, which is 102.
After making the sites.dat changes, I did not need to use the Setup.py script, but I did need to use the Citi 10-minute process to relink the PocketSense scripts to my Citi account. The easiest way to do that is to log in to https://citi.com/datamanagement.
In looking at the downloaded OFX file, I saw that it said VERSION:102, so I tried removing ofxVer : 103 from sites.dat, but then I received this message in the OFX file: "We've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement."
I then put ofxVer : 103 back into sites.dat, and the downloads started working again, so that was good enough.
How this worked for me was that after going to https://citi.com/datamanagement to enable access I then did a PocketSense Getdata.py transaction download during the 10-minute setup period. That transaction download worked normally, and after that the Citi transaction downloads continued to work normally.
Other people reported needing to use PocketSense Setup.py to delete and recreate their Citi account definition, which I tried before making the sites.dat url and ofxVer changes, but that did not work for me. Then, after making the sites.dat url and ofxVer changes, it was no longer necessary for me to use Setup.py to get the transaction downloads working.
A few months ago, I had a similar problem with American Express transaction downloads, and for that I did need to use PocketSense Setup.py to delete and recreate my American Express account definition. I also deleted and recreated my Citi account definition at that time because I was using an out of date account number that I was afraid would stop working. That was a pain because I ended up with duplicate transactions downloaded back to the beginning of 2019, so I did not particularly want that to happen again.
People who use Moneydance are reporting problems getting their AT&T Citi downloads working, so maybe there is something different about the AT&T Citi cards. Here is a link to a Moneydance forum where this is discussed:
Could have been the complaints from the Moneydance people.
Could have been that I enabled access before setting up the account in PocketSense, instead of first setting it up, then enabling access, and then testing it.
Wow, lots of excitement since i was last able to log on to the blog.
- My Discover downloads are still broken, but as was mentioned above by myself and @redmondman Discover downloads into Quicken ARE working. They go through a proxy-like setup though, rather than directly to ofx.discovercard.com - Discover customer service was so far useless "contact your download service provider". They don't understand the service provider is ME (well, really Robert's scripts, but I don't wan to confuse them). - Fidelity, Chase, BoA are not having trouble so far. I do not have Citi
Robert - I had written a pretty nice ingestion script for downloaded statements. I've been using it for months to processTarget and Transamerica, but haven't cleaned it up for broad distribution. I will send your way to see if it adds anything to what you've come up with.
I have used Chase and Discover for decades with OFX and now with Discover dead I will start using just my chase visa but what bank options do I have?
I was ING Direct forever then then they were Capital one 360 then they stopped OFX and I changed to Discover bank, now they have stopped OFX so I need a new high interest saving bank that has OFX support? Suggestions please?
Heh. I've been a credit union member for decades, so I don't have a good answer. I've adapted through OFX outages, changes that required workarounds, and have finally resorted to custom code to work with my credit union (First Tech Federal).
We're a dying breed; I just think it's going to keep getting harder.
" now they have stopped OFX so I need a new high interest saving bank that has OFX support? Suggestions please? "
Ally has high yield savings, but only has web OFX file download for savings. Still, it's better than most. No OFX for CDs.
I observe people posting about successful PocketSense access to Chase Bank, I think. But I cannot be sure because when they say Chase, without saying bank or credit card, who knows? Also, they will not meet your high yield savings need.
Your best bet would be a broker, and put you money into a premium MMF. They pay around 2% and you can get check writing. Several have good OFX via PocketSense.
There are times when you actually need a local bank. A local broker's office is not the same.
In my experience, you can tell a lot about a bank (and most other service companies) by simply watching how they handle their clients.
For example, American Express went through one of these conversions that affected OFX processing a while ago, and they were on top of it all the way through, telling you, in advance, exactly what you had to do -- and it worked exactly the way they said it would. In contrast, Capital One was completely brain-dead when they killed off OFX, and I closed all my accounts there because I viewed it as the "final straw".
Discover has always had pretty poor customer service, so I am not surprised by their left hand having no idea what their right hand is doing.
Nothing to add, but your comment about CapOne made me want to signal my agreement. Had been a customer since ~2005 and I finally switched out after the OFX and the requirement to have a mobile number (they were always trying two factor auth and I don't do that over text, nor do I want to give any company my mobile number). I can add that I setup CITI cards today. Not sure if I had to delete them, but I deleted the accts in pocketsense, then started the online 10min timer and re-added them. They were doublecash (mine and SOs). I had 103 and the mobilesoa site in the sites.dat file. Hopefully it continues to work without issue.
Discover seems to still be dead in the water...waiting on that one until I see more comments from quicken or our community.
Barclays I believe is a no (at least credit cards). I haven't looked into it for years, but I don't think they do direct connect. I have to login and download my transactions manually every month.
2. Logged into Citi account center for AT&T card at https://www.citi.com/credit-cards/citi.action 3. Selected: Profile-->More Settings-->Manage Desktop Apps-->Add access, which started a 10 min window 4. Setup.py: * Delete existing account. * Add new account using above site. * Select account number given in list. The account# (literally) looks like XXXXXXXXXXXX1234, w/ only 4 digits. * Test == success * Option 7: Test... success again.
When you send to Money, you'll need to link the new account# to the existing Money account.
To the best of my knowledge no one has gotten downloads to work for Citi savings or checking accounts. I have a checking account so if you figure out how to automatically download using PocketSense please post it here.
Even the Citi bank OFX downloads from the website have become deficient compared to how it was a few months ago. There are FITIDs not staying the same for the same transactions downloaded later, and only one account can be downloaded at a time.
Idea for scrubber element, to be called within scrubbers:
If there is a <NAME> in a credit card transaction, consult a list of RegEx. With the first match, if any, replace the <NAME> field with the corresponding entry. Then create a <Memo> with the original Name content if there was no Memo already. If there was a <MEMO>, concatenate to the content up to a 255 character max.
Thus THE HOME DEPOT #2036 JOHNSTOWN THE HOME DEPOT #3917 SMITHVILLE THE HOME DEPOT 130 THE HOME DEPOT 2036 JOHNSTOWN THE HOME DEPOT 3917 SMITHVILLE
etc could all become "THE HOME DEPOT" or "HOME DEPOT", as you wish. The store info could be put into a <MEMO>
This would only affect new transactions, but you could consolidate in Money.
Even worse is Amazon, where payee is unique for every transaction. I select my generic "Amazon.com" when accepting the downloaded transaction, but if the scrubber would do this, I'd like Cal's suggestion.
Put this near the top of scrubber.py (after the "stat" line is fine):
def _scrubAmazon(ofx): # 1. Scrub the trailing "*M12345678" from Amazon.com purchases on the Amazon Chase Card p = re.compile(r'<NAME>.{0,12}\*[0-9A-z]{9}([<\n])',re.IGNORECASE) if p.search(ofx): ofx = p.sub(lambda r: '<NAME>Amazon.com' + r.group(1), ofx) scrubPrint("Scrubber: Amazon.com order numbers removed for aliasing.")
return ofx
Put this just above the DISCOVERCARD and TROWEPRICE sections of the scrub routine in scrubber.py: if 'CHASE' in siteURL: ofx= _scrubAmazon(ofx)
I use this on my Chase Amazon.com card. You can adjust the location of the scrub if you use multiple cards for your Amazon purchases. The REGEX finds a name with 0-12 characters, a *, and then exactly 9 letters/numbers. This matches Amazon.com, AMZN Marketplace, etc. I haven't had a false positive yet. You could make it more specific to prevent that from occurring.
Andrew , NxtTek, and all: I think for a function used by non-programmer people, that the data for a function might be entered something like this: PS_payee_meld_table=( ("Amazon.com","Amazon"), #if it starts with Amazon.com, change to Amazon. ("AMZN Marketplace","Amazon"), #similar ("THE HOME DEPOT","HOME DEPOT") #Consolidate Home Depot payees. )
The ^ at the start of the string could be done by the function rather than the user. Where the desired action is to switch if it contains, then enter the something like ".*wendys" and so on.
Or the data could be read in from a csv file rather than a Python file, so the user could prepare the list with a spreadsheet if desired. MSMoneyQuotes does it that way for a lookup table. Perhaps pass the name of the spreadsheet as a parameter. My concern would be if a payee contains a comma.
I would propose that the function break out the Name field contents, and each name be compared to the lookup tuple until one is found or nothing is found. In this way, no bad RegEx could do bad things other than cause a mismatch. Andrew, you have the knowledge to get a RegEx right, and the use that in a sub(). RegEx are darned easy to get wrong, so let any mistake only cause no action or a wrong payee entered. Limited consequence.
I've been using PocketSense for years and I just ran into a new problem. GetData.py does not work (does not request password and says, "No accounts have been configured. Run SETUP.PY to add accounts.") ofx_config.cfg is present, populated, and unchanged from several months ago. Running Setup.py does not request password and shows no accounts. sys.version_info(major=2, minor=7, micro=8, releaselevel='final', serial=0). Any suggestions?
TD Bank - I was able to get the TD Bank download to work, couple of issues that are dealt with to setup
First - the settings SiteName : TD Bank AcctType : BASTMT #bank fiorg : CommerceBank fid : 1002 url : https://ofx.tdbank.com/eftxweb/td.ofx bankid : Required – enter routing number ofxVer : 103
Second - run setup After entering username and password you will get notified of an error, now just leave setup open and running
Third - after about 1 minute log into the TD website for your bank account Go To Account Options-Account Services In the manage linked services box, under financial tools, click Manage Settings Accept/Approve Quicken 2015 If necessary, log out and log back in to try again and get approve option
Fourth - Go back to setup and say yes to configure manually Enter account number Choose account type At this point the download should work (I cant remember if you have to say yes to test, yes to send to money or yes to both, but it should work)
I thought I'd gotten TD Bank to work following Michael's method, but it isn't. I'm getting through the first check at TD Bank but then there's some kind of hiccup. If anyone can figure out what's causing the "Duplicate Request" error I'd greatly appreciate it.
Ken, from this site http://libofx.sourceforge.net/html/fx-0_86_85_2lib_2ofx__error__msg_8hh-source.html Duplicate Request = "Duplicate request", "A request with this has already been received and processed."
Maybe try closing setup and retrying, but skipping the part about TD website as that is now activated. In setup after entering username/password and you get the error, say yes to configure manually and enter account number, etc
If necessary you can delete the approval from the TD website and start over. Not sure exactly how I got it to work
The error message got altered, here it is again without the problematic brackets "Duplicate request", "A request with this TRNUID has already been received and processed."
Are you using the most recent version of the scripts? The TRNUID is unique for each request. However, I don't recall if a static value may have been used in old versions (?).
A quick question for the forum - not really a Pocketsense query but a MSMoney one.One of my financial accounts only offers downloads as a Quicken file (.QFX or .QBO). Until today I have been able to download a file in QFX format,change the extension to .OFX and then Moneyhandler would process the file happily. As of this morning I am getting a message from Moneyhandler saying the file is either invalid or contains corrupt data. I'm assuming Quicken have made some sort of change to their transaction formats and wonder if anyone has found a way to continue to import their files into MSMoney? Thank You
I download QFX files from Discover Card. I don't bother to rename them to OFX though. I just change the 'Files of Type' drop-down to 'All files (*.*)' in the import window. The QFX file imports as expected.
Robert's latest dev/test version does a great job of importing the files automatically, if saved into the import folder. It takes care of the extension for you, as well, so just leave it as .qfx
> Discover seems to still be dead in the water...waiting on that one until I see more comments from quicken or our community. The Http Header Host needs to have a port number. So try using the following the setting in the sites.dat file
hleofxquotes and NxtTek: Success is good! But mysterious in that 443 is the default port for HTTPS. I think there must be a quirk somewhere, and this works around it.
I didn't try Discover this morning with the standard sites.dat entry. Perhaps that works too? I just tried the suggested change and "discover"ed that it works ;)
NxtTek: if you tried, and it did not work, as an experiment, you might try url : https://ofx.discovercard.com? WITH the question mark. The theory would be that maybe the original URL got cached. The ? would make the string different, and would not use the cached version.
Before posting this, I tried adding a question mark at the end of the Vanguard URL; that worked in a test.
Thank you for posting. I confirmed that it works for both Discover Card and Bank. For anyone curious, the https connection uses port 443 (as Cal mentioned), but explicitly putting it in the URL means that it also gets included in the HOST header in the http/POST request. Go figure as to "why" it changed...
Has anyone used the .csv and .json to .ofx converter from: https://github.com/kedder/ofxstatement ? I opened an account at Simple Bank and they only provide downloads in .csv and .json formats so I went looking for a converter. Is anyone using a converter that's easy to use and straightforward? The one I found at github requires Python 3 and I don't know if there'll be a conflict if I have both Python 2 installed for PocketSense and Python 3 for this app.
Ken: https://www.activestate.com/products/python/python-3/ says " Since you can safely install both ActivePython 2.7 and ActivePython 3 side-by-side on your computer, you can experiment with Python 3 while still using Python 2 for your current code."
Provided that you don't mind using qif instead of ofx, take a look at xl2qif. It's an add-in for Excel. As the name implies, it converts csv to qif. It works well for Bank and Credit Card transactions. It's free to use and easy to set up. http://xl2qif.chez-alice.fr/xl2qif_en.php#download
@Kevin Thank you! Already downloaded the Add-in and figured out exactly how to use it. It's pretty spiffy and will make it much easier to keep Money up to date for those institutions that don't provide a QFX or OFX download capability. Really useful.
@Cal Thanks for the Python info. The Add-in that Kevin suggested will probably do the trick. If I have some free time over the holidays I'll try and get the Python OFX converter to work and see if it imports more robust detail. But for a quick and dirty, the Add-in definitely makes things easy.
I cannot believe discover works again! I changed everything to Amex after 20+ years of downloading discover transactions. And then just this morning I deleted all my discover bank accounts (6 of them)... I was unable to find an alternative bank that worked with pocketsense that was high interest rate, I guess now I don't have to! I actually came here to see if anyone had posted if anyone had any other banks. BTW discover told me...
"Due to a recent update to our website, we can only accept OFX connections from Quicken, QuickBooks, and eMoney. Other financial software will no longer be able to connect. I apologize for any inconvenience this causes."
Ok I set everything back up and did find a problem with discover card's ofx server. They must have some sort of control on it based on IP or something to prevent multiple hits in a row. When I set all the accounts up and test them (with a few minutes in between) they all worked fine. I then used the script to download them all and the first 3 (of my 7) worked. The final 4 got access denied. I then waited a minute and manually checked one of the ones that was denied through the test in setup and it worked fine again. I then ran the entire script again and only the first account worked the other 6 came back with access denied. Finally I ran the whole thing again right after and all 7 accounts were denied.
I have played a little with this but can someone else test making a request a few times in a row and see if they temporary ban you like it appears is what is happening to me?
I have 2 Discover Card accounts. Getdata.py successfully connects to both of them without incident. Perhaps there is a delay-timer that can be set within the scripts to prolong the time between connection attempts.
Kevin can you do me a favor. Go into the setup and test the account several times in a row and see if it eventually gives you the "You don't have permission to access" error. Yes I think the delay might work great, I just want to verify that is in fact what is going on. Thanks
Hi Anonymous, I ran through 'Test Account' several times for 1 of the DC accounts without incident. Then again alternating between the 2 accounts. Again, without incident. -Kevin N.
Thanks for checking. I just tried it and it did 5 of the 7 before it would not allow it again. Maybe I will add some fake account or something between them in the script that will need to timeout in order to cause some delay because it does certainly appear to lock me out after hitting it enough times in a row.
Is there an easy way to change the order that it checks the sites in? Trying to mix my discover bank accounts in with the other accounts to hopefully get past this blocking issue. If not an easy way to add a delay? thanks
Thanks, so I can add more discovers to sites.dat however when I check them it will always cycle through them in the order I added them in setup (or the order that I see if I list accounts in setup), not in the order the banks are in sites.dat. I have 14 accounts that I have pocketsense check (and then investments) and it does those accounts in the order I add them. I was trying to avoid just going and and clearing them out and readding in a different order. Wondering if there was a cfg file or something I could just edit and change the order.
If you sort your xfr folder by date modified, I think you'll see that the download order is alphabetical. Same with the combined file, if I'm not mistaken, which is why I made the recommendation of "A DISCOVER & Z DISCOVER".
Thanks again, but mine does seem to be doing it by order they are listed in the setup. My xfr folder goes etrade, Vanguard, Fidelity... and so on the exact same order as it downloads them and shows them in the list.
I was able just to add a few bad login etrade attempts between the discover attempts and that slowed it down just enough that it no longer gave me a security issues and downloaded all 7 accounts. Thanks
So... you only need a second or so between attempts? I need to get the current test version posted, and could add a delay option, prob at the Site level? The idea there would be to have a user-defined delay to insert before attempts to any account defined using that site entry. Would that help?
As for account order, it's not supported. Account entries are "pickled" into a linear/flat file named ofx_config.cfg as they're created. While much of it is encrypted, there's a distinct storage pattern, and "blocks" could be moved around to change order. It would be a hack, so if done wrong, would break it.
I could probably (easily) add a simple sort option, if needed. Something like "sort alphabetically by site name up or down". Useful?
The fake account trick between my discover accounts is working great. Sure a delay I could set would also work and look nicer but in the end it does work and if I'm the only one with the problem I would not worry about it. Thank you again for this software! When discover was out I looked at the alternatives again and NONE could do anywhere near what good old MS money can do (along with pocketsense).
I use a Windows scheduler to run GetData.py early mornings Tuesday through Saturday. The reason is to not get quotes imported into Money investment accounts for Saturday or Sunday. Quotes are a big factor in how big a *.mny file grows.
If there were a command line switch to let me restrict or specify accounts to be downloaded, I could invoke with the switch to get the credit cards over the weekend. I don't even know if they get posted over the weekend. If this were a bigger deal, I could use two different PocketSense folders, and use one or the other depending on which accounts I want for that day. Not a big enough thing for that for me.
Has anyone else experienced a drastic slow down in quotes for mutual funds and/or stocks within the past few days? My quotes are taking about one minute each to process and I obtain +/- 50. First noted on Thursday, Nov. 14th.
Robert, thanks for starting a new thread for comments; the "load more" was definitely causing a problem.
ReplyDeleteFor those who discover the issue with downloading for AMEX credit cards, please refer back to the previous thread which can be found at the posts beginning with packpike on August 8, 2019 at 10:15 AM near the bottom (after "load more") at https://pocketsense.blogspot.com/2018/08/new-version-available.html
There are also loads of other topics addressed in that thread so you can search there before posting your issue :)
The "load more" does not work for me. The last comment I see is from Joe@TheWho.orgMay 29, 2019 at 7:37 AM.
ReplyDeleteAny way to get to those comments after May 29? Any way to post the AMEX card fix here? I think I may have mistakenly asked multiple times because I couldn't load more comments.
Load more should be an active link underneath the comment box at the bottom of https://pocketsense.blogspot.com/2018/08/new-version-available.html
ReplyDeleteI just checked it and it still works. If it's not active for you then it's something on your system.
Here's what I previously posted:
Ken August 8, 2019 at 10:39 AM
You're not alone. I tried to update my Amex accounts and ran into the same problem. Figured it was a temporary glitch so didn't pay much attention. Thanks for taking the next step and finding out it's a system change on Amex's side. The solution for us is actually quite simple. Just delete the account from PocketSense and then add it back in. You'll notice that when you enter your login ID and password to add the account, the list of your Amex accounts will look quite different than they used to. Similar to Discover, they're now using non-account numbers.
Make sure you're using the most recent PocketSense or it won't work.
Ken
DeleteI think there's something wrong with my system (preventing that javascript code from running to load more). I was able to use another computer to see the rest of the comments.
Thank you for reposting the fix.
R
ReplyDeleteMicrosoft Money Sunset Edition still works fine on my Windows computer. I've been using Microsoft Money for many years and it's like a comfortable old shoe. However, I've gotten worried lately. I've invested many thousand of hours of my life tracking my personal financed in Microsoft Money, I have transaction data in Microsoft Money all the way back to 1992. It all sits in a proprietary data format accessible only with software that was abandoned by Microsoft years ago.
I check out all the recommendations I could find for MS Money replacements, and they all had proprietary data formats. Then I found Tiller ( tillerhq.com )
Tiller uses a Google Sheets spreadsheet for transaction data and does really slick and automatic updates from my financial institutions. Since the data is all in a spreadsheet format, it is "future-proof" and not in danger of loss like proprietary Microsoft Money data. I've been using Tiller for a month now, and the more I use it, the better I like it.
The process of getting my financial life going back to 1992 moved over to Tiller-compatible spreadsheets was non-trivial. I probably spent over 8 hours doing this. I had dozens of old credit card accounts, 9 checking accounts, and many investment accounts in Microsoft Money that all had to be exported and converted, one by one. As an aid to myself during this process, I wrote a cheat-sheet. Today I updated my cheet-sheet into a full step-by-step instruction for this process. It resides in a Google Sheets spreadsheet and here's the link
https://docs.google.com/spreadsheets/d/11N5Iz_Hqz-JABeSVTXF2SttXHyfnvZXcACAl81s_7vs/edit?usp=sharing
This alternative to migrating away from Microsoft Money has the advantage of being accessible to fairly non-technical people. I just thought you might be interested.
Cheers; Art King
Anyone having a problem with Amex downloads - running into this for the past 2 weeks - no changes in system. The output I get is (note I had to replace CODE with "----" to allow this post): <---->0INFOLogin successful20190831075541.301[-7:MST]ENGAMEX3101FMPWebBCM201908310755415ceedf48-f311-4a4d-9c44-297403c2a1d1<---->2003ERROR4
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteSorry, just saw the first Post (forgive me - first time here - non techie, just worried) about Amex. Thanks so much.
ReplyDeleteI manually download and import QFX files from TD Bank credit card. They updated their website a few weeks ago and since then Money can't import the file.
ReplyDeleteTurns out the file now includes pending transactions with an empty FITID, and transactions with empty SIC fields that Money can't process. Also all debit transactions have positive amounts, so Money interprets them as credits.
I added the following regexes in combineOfx to fix these issues:
```
# remove all pending transactions with empty FITID
c = re.sub('(.*?<.*?)', '', c)
# remove empty SIC lines
c = re.sub('<', '<', c)
# make all debit transactions negative so Money doesn't treat them as credits
c = re.sub('(DEBIT.*?)(\d)', '\g<1>-\g<2>', c)
```
I use something similar to the following script to process and import manually downloaded files:
```
import os
from control2 import *
from rlib1 import *
ofxfiles = ["td-visa.qfx", "fidelity-visa.qfx"]
ofxpath = "c:\\ofx\\"
ofxlist = []
for arg in ofxfiles:
ofxlist.append(['', '', os.path.join(ofxpath, arg)])
cfile = combineOfx(ofxlist)
runFile(cfile)
```
Roman: Your code to remove transactions meeting a criterion is a whole lot simpler than mine. I suspect two things:
ReplyDelete1.When you post, the forum removes things between < .... > and big pieces of your expressions get removed. If you replace all < with < and > with > then that problem will not happen. Some replace those characters with curly braces {} for posting purposes.
2. Simple expressions often match a lot more than you think and can remove more than one transaction. Watch for that to occur.
cl
1. Ah, you are right, didn't notice it last night. Too used to the convenience of triple backticks in Slack and such. Reposting with brackets properly escaped.
ReplyDelete2. I agree that regex can be quite dangerous. I used greedy matching, but it sure can break if transaction tags are not closed properly or if fields in a different order. The proper way is to split the file into transactions and then check each transaction separately, but I was just happy to get it to work. As the saying goes:
"Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems."
Regexes in combineOfx to fix TD Bank OFX:
------
# remove all pending transactions with empty FITID
c = re.sub('(<STMTTRN>.*?<FITID><.*?</STMTTRN>)', '', c)
# remove empty SIC lines
c = re.sub('<SIC><', '<', c)
# make all debit transactions negative so Money doesn't treat them as credits
c = re.sub('(<TRNTYPE>DEBIT.*?<TRNAMT>)(\d)', '\g<1>-\g<2>', c)
------
Process and import manually downloaded files:
------
import os
from control2 import *
from rlib1 import *
ofxfiles = ["td-visa.qfx", "fidelity-visa.qfx"]
ofxpath = "c:\\ofx\\"
ofxlist = []
for arg in ofxfiles:
ofxlist.append(['', '', os.path.join(ofxpath, arg)])
cfile = combineOfx(ofxlist)
runFile(cfile)
------
My CitiCard account is the first time I had to go to OFXver 103, which I think involves the use of ClientUID. Then with the Citicard website, you click to a place that starts a 10 minute window where a new desktop client can get in.
ReplyDeleteI am thinking the ClientUID gets remembered during the 10-minute window, and then you don't need to use the web access subsequently. I think the ClientUID is a bit secret -- should not be posted. Other things in the sites.dat file are not at all sensitive, and before ClientUID, that file could be freely shared. I am thinking that the ClientUID would have better been put into the always-sensitive/secret ofx_config.cfg file. Of course the ClientUID is just one factor in a 2-factor authentication method. Just a thought. I am very happy that the CitiCard transactions download without drama now, and I can remember to consider my sites.dat sensitive.
CL
OK, I think I missed out on paying attention to ConnectKey, which is in the ofx_config.cfg file. I expect that ConnectKey is the important one.
ReplyDeleteCL
Hi All,
ReplyDeletePython 2.7.14
PS 24-AUG-2018
Win 10 Home 64-bit
Is anyone else having issues with Discover Card?
For the past few days, I'm receiving an 'Invalid OFX statement received' error.
The contents of the OFX file are as follows... (replaced < and > with [ and ])
[HTML][HEAD]
[TITLE]Access Denied[/TITLE]
[/HEAD][BODY]
[H1]Access Denied[/H1]
You don't have permission to access "http://ofx.discovercard.com/" on this server.[P]
Reference #18.a2881ab8.1568292469.dcf70cb
[/BODY]
[/HTML]
-Kevin N.
Same. I have tried some alternates from MoneyDance with no success. It appears that ofx.discovercard.com is not accepting connections anymore.
DeleteHi Andrew,
DeleteThank you for your confirmation.
I sent a message to Discover Card support. I'm not expecting much but it's worth a shot.
I'll post back if I hear anything from DC support.
-Kevin N.
I'm having the same problem so reset the connection. That didn't help so I deleted and tried to re-add the account. Here's what I got: An error occurred requesting accounts from the site. Please check username and password.
DeleteI'm guessing either their OFX server is down or they changed the URL. Will try and re-add the account in a couple days. If either of you start getting downloads again, please post a message here.
Thanks.
Hi Ken,
DeleteThank you for your confirmation as well.
If I remember correctly, Robert is a DC client, so we might be hearing from him on the matter.
I will post here if things start working again.
-Kevin N.
Just to add that I am having the same problem with both Discover Card and Discover Bank. Wouldn't Quicken be having the same issues?
DeleteHarold
They are indeed having problems, although I don't understand their terminology, so I can't figure it out. https://community.quicken.com/discussion/7858430/discover-card-and-discover-bank-direct-connection-error-ol-294-a
DeleteHaving same issues with Discover, although I started having them a bit later than those in the Quicken thread. Quicken and Discover pointing fingers at each other. Some people in the Quicken thread indicated that they had to update their "Financial Institution Branding and Profile". Sounds like they are either changing servers or login methods, but not clear how yet.
DeletePer a post on the Moneydance forums, Discover is no longer supporting non-Intuit clients.
Deletehttps://infinitekind.tenderapp.com/discussions/online-banking/14201-unable-to-download-discovercard-transactions/page/1#comment_47655016
It seems that the latest preview version of Moneydance MD 2019.4 (1892) is able to connect to Discover Card.
Deletehttps://infinitekind.tenderapp.com/discussions/online-banking/14201-unable-to-download-discovercard-transactions#comment_47804342
Discover update. Still doesn't work in Pocketsense. I tried setting up in my test Quicken application. Set it up as Direct Connect, not Web Connect. I look a the logs, and there is definitely interaction with ofx.discovercard.com. First attempt with Quicken didn't work, but second downloaded just fine.
ReplyDeleteSome more information for those more techie minded. I did some packet tracing. When PocketSense goes out it queries ofx.discovercard.com directly, which is really an akamaiedge.net cloud server.
DeleteWhen quicken claims to query ofx.discovercard.com (according to its own OFX logs) it first talks to services.quicken.com and then to a completely different akamaiedge.net server.
Again this is using Quicken's direct connect. So one thing is clear, Quicken is doing something different. I don't know if it has dedicated servers on the Discover, or if it is using its own proxy like system or what. I haven't tried something else like Moneydance.
Carlos: Could send a full trace log of the Quicken connect, sanitized of personal info to pocketsense1 at gmail dot com? It may be that the first interaction was key also, so if it's possible to remove the account in Quicken and add again and get logs for both connections, then that may be better.
DeleteCarlos, I think all Quicken queries first go to their servers, so that they can verify a current paid subscription for INTU.BID 9625; INTU.USERID xxxx. You may be correct though, in that an entirely new url may be in use for Discover, that is unknown to us.
DeleteI should have added... http connection headers will likely be the key parts.
DeleteHm, I still have an active Quicken subscription for troubleshooting. In the options for "connection setting", I see "Express Web Connect" and "Web Connect"; I don't see an option for "Direct Connect". I wonder if there's a way to set "Direct Connect" that I don't see. If I can get Quicken to work, I'll happily send over some HTTP(s) connection headers.
DeleteRedmondman, https://community.quicken.com/discussion/comment/20029957#Comment_20029957 comments by digitalmediaphile may get you to Direct Connect for some log-worthy action. I don't follow the steps, myself.
DeleteCL
Fiddler tracing adds to what Carlos has mentioned. Quicken does not connect directly to ofx.discovercard.com, but instead via a proxy. This hints at some serious, OAuth-type authentication that may be difficult to duplicate. The rest of the headers look standard. You'll recognize 7101 as the FID for Discover, of course.
DeletePOST https://services.quicken.com/ofx-secure-plus/7101 HTTP/1.1
Authorization: Bearer [a nice long auth token]
Just to corroborate Carlos, for Citibank (by contrast, for example), CONNLOG.TXT shows that Quicken is talking directly to 'https://mobilesoa.citi.com/CitiOFXInterface'.
DeleteVs. Discover, where CONNLOG.TXT shows that Quicken is talking to 'https://services.quicken.com/ofx-secure-plus/7101' and never directly to 'https://ofx.discovercard.com'.
I'm curious whether anyone knows if other financial software besides Quicken is working with direct Discover downloads? I only use PocketSense with MS Money so don't know if we're the only ones still experiencing issues. Does anyone follow support issues for other apps?
ReplyDelete@ken, I use Moneydance and it can successfully download Discover transactions. On a semi-related note, many people are having problems using Moneydance to download Citicard transactions, which is why I'm testing Pocketsense.
ReplyDeleteHa, time to play with Moneydance again. I'll report back tonight if I find anything worth reporting.
DeleteLarry, I was able to get Citicard working with Pocketsense. Follow https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/comment-page-2/#comment-11241 to see how. It's Discover that has stopped working for us after all these years.
Delete@Larry,
DeleteCan you confirm that Moneydance is indeed connecting to Discover Card?
I'm using MD 2019.4(1886) which currently is NOT connecting.
When attempting to connect the DC, MD does not display the usual visible error message window. Instead you need to view the contents of the console window to see that the connection failed.
-Kevin N.
Moneydance is having problems with Discover as well. Discussion thread is here:
Deletehttps://infinitekind.tenderapp.com/discussions/online-banking/14201-unable-to-download-discovercard-transactions
Console log shows me:
uh oh, we've gotten some OFX with no headers: [HTML]
Unhandled OFX message set: HEAD
Unhandled OFX message set: BODY
uh oh, we've gotten some OFX with no headers: [HTML]
Unhandled OFX message set: HEAD
Unhandled OFX message set: BODY
account: Discover; serviceType: 5; service: Discover Card; mode=both
Any tips on how to run the Discover scrubber independently? We can download the OFX file from the web site, but the file still needs to be scrubbed.
DeleteMy Bank of America (BofA) accounts stopped working, both checking/savings and credit cards. The ofx file says "Invalid username/password combination has been entered. Please try again or sign-in to bankofamerica.com to restore your connection from your computer." I updated to latest PocketSense and Python, still same issue. I read the thread from 2017 referencing this issue, but none of the solutions work. My sites.dat entry:
ReplyDeleteSiteName : BANK OF AMERICA
AcctType : BASTMT #bank
fiorg : HAN
fid : 5959
url : https://eftx.bankofamerica.com/eftxweb/access.ofx
bankid : 011000138
ofxVer : 103
appid : QWIN
appver : 2400
No problem with BoA on my side, although Bank ID is different. I have two different accounts with bankID 052001633 and 051000017. Both working. I leave appid and appver empty
DeleteI've had no problem with either BofA checking or credit card account downloads. Here are my sites.dat entries:
DeleteSiteName : Bank of America
AcctType : CCSTMT #credit card
fiorg : HAN
fid : 5959
url : https://eftx.bankofamerica.com/eftxweb/access.ofx
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval :
SiteName : BofA Checking
AcctType : BASTMT #bank accounts
fiorg : HAN
fid : 5959
url : https://eftx.bankofamerica.com/eftxweb/access.ofx
bankid : 081904808 ***This should be the ABA routing number off your checks***
brokerid :
appid :
appver : 2400
ofxver : 103
mininterval :
Be aware that for checking the BankID should be the ABA routing number shown on your checks. If you still can't download then I'd delete the accounts from your account list and re-enter them using just the login ID and password to get the account list from BofA.
redmondman:
ReplyDeleteExample using Fidelity scrubber. Discover scrubber is different because it
has second (accType) parameter. Adjust code accordingly.
This file requires no indent. Presumes scrubber.py is in the same folder.
Parameter from command line is path to downloade OFX file.
===================begin python file====================
#run scrubber against ofx file. Write modified file.
import scrubber as s1 #can then invoke s1._scrubFidelity(ofx) for example.
#ofx_mod=s1._scrubFidelity(ofx)
import os, sys, glob, time, re
import win32gui as w32
from win32con import *
ofxfile1= open(sys.argv[1],'r') # open for read in current directory
# Input file read in as parameter. Output file fixed in this case.
ofxfile2= open("ModOut.ofx",'w') # open for write in current directory
#do stuff
ofx=ofxfile1.read()
ofx_mod=s1._scrubFidelity(ofx)
ofxfile2.write(ofx_mod)
ofxfile1.close()
ofxfile2.close()
print "Modified version of ", sys.argv[1],' written to "ModOut.ofx" file.'
==============end python file==================
CL
Thanks for your stand-alone version! I have an OFX file from Wealthfront where they use BUYSTOCK for the cash sweep account but it's actually a mutual fund and should be BUYMF (the SELLs and INCOME are appropriately MF entries). I wrote a small REGEX that fixes it but had to manually copy/paste the file into a REGEX parser. Now I'll write a quick script that runs it automatically.
DeleteThanks!
Andrew: for your sweep, you could just remove the Buys and Sells for that MMF. I have some code made for that.
DeleteThen the cash balance for the account reflects the net MMF buys and sells.
CL
I have thought of doing that before but I like the feature of seeing the investment performance of that sweep account in the Investment View.
DeleteMy scrubber:
def _scrubWealthfront(ofx):
# Change the bad BUYSTOCK entries to BUYMF
p = re.compile(r'<BUYSTOCK>((?:.*\n){1,10}\s*<UNIQUEID>74926P688<\/UNIQUEID>(?:.*\n){1,12}\s*)<\/BUYSTOCK>',re.IGNORECASE)
if p.search(ofx):
ofx = p.sub(lambda r: '<BUYMF>' + r.group(1) + '</BUYMF>' , ofx) scrubPrint("Scrubber: Wealthfront BUYSTOCKs fixed.")
return ofx
Also having problems with discover card and discover bank ofx downloads for last week or so. However it is even more than that. I cannot log in to the app or website either. I called them up and they opened my account to check it and boom right then on the phone app and website worked again. They said everything looked good... I hung up and player around a bit then tried to download ofx files again and right away app and website stopped working and would not take my login again. I called discover back and they opened up my account again to check it and again boom both app and website started working again, I said "I wonder...", tried to download statements again with pocketsense and boom instantly locked out again from app and web site.
ReplyDeleteI told them all this on the phone but who knows if this info ever gets to the right people. But not only can I not download my statements it hoses my account so I cannot log in.
Interesting. I haven't had that issue. I've tested w/ Discover/ofx *many* times over the past few days, and my app and web login work fine.
DeleteOk I appears to be time based. If I attempt to use my normal script to access my 1 discover credit card and 7 discover bank accounts I am locked out of my app and web login for appx 15 minutes then all is back to normal. Its strange but true. I would guess it is some sort of security trigger I am setting off.
DeleteI added some functionality over the weekend to support importing downloaded ofx statements via the getdata.py script. While this isn't needed for some sites, some (like Discover) need the additional "scrub" ;) For anyone interested, a beta/test version has been uploaded for testing.
ReplyDeleteA few notes about how it works:
1. Setup now creates an ./import subfolder (alongside ./xfr, etc). Running setup creates the folder, but you can add manually too.
2. Any ofx, qfx, etc. file can be dropped into folder. No need to rename.
3. When Getdata.py is run, it will process *all* files in ./import, and if it "looks like" an ofx file, it will be treated the same as a downloaded file (scrubbed, combined, etc).
4. If accepted as a valid ofx file, it is moved to the ./xfr folder (i.e., it is consumed).
An attempt is made to figure out what sites.dat site entry to use by looking for a matching FID or BANKID value in the statement. If either of those are found and match an entry in sites.dat, then the site is selected. This matters when applying site-specific scrub routines.
There are a number of possible issues with importing a downloaded file in this manner, but it should work for most cases. The oddest I've seen is a manual Citi CC download not including the account number, but instead looking something like "xxxxxxxxxxxxxxxx1234".
For DISCOVER credit card users, manually downloaded statements use a different FID value than the direct-connect version. For now, I've deleted my Discover accounts in Setup, and added the following entry to sites.dat to correctly match to a manually downloaded discover CC statement. The Discover *bank* entry is the same for both the ofx and manually-downloaded statements.
<site>
SiteName : DISCOVERCARD_Import
AcctType : CCSTMT #credit card
fiorg : Discover Card Account Center
fid : 9625
url : https://ofx.discovercard.com
</site>
I'll await comments/issues before posting for general use.
- Robert
To clarify... my existing Discover Bank site entry works correctly with an imported statement. Only the credit card settings/entry are different.
DeleteHi, Robert...
DeleteDo we have any idea why the "regular" Discover downloads aren't working? And if there is a prospect of getting them to work again?
Harold
Discover changed something on their site that's breaking PocketSense, Quicken and Moneydance. I'm guessing we'll need to wait for reports of success for at least Quicken before Robert (or any of us little helpers) are able to assess PocketSense. In the meantime, this is a handy convenience for those of us working around the issue by downloading the QFX files from the website.
DeleteJudging from observation, Discover is now blocking access to ofx.discovercard.com to all entities except for those they have an OAuth trust relationship with. It's a good security measure, and big guys like Quicken will be able to make it work (once they work out the kinks), but if I'm being pessimistic, it may be the end for homebrew scripts like PocketSense.
DeleteIt may be way worse than just Discover. I'm now getting the same "Access Denied" from Fidelity. If I can't fix this I'm completely screwed as that's where 90% of my transactions take place and there's no way to initiate downloads from their website.
DeleteI can confirm, same error message from Fidelity starting this morning.
DeleteHappening here, also, with Fidelity starting this morning.
DeleteError in OFX file:
Access Denied
You don't have permission to access "http://ofx.fidelity.com/ftgw/OFX/clients/download" on this server.
Reference #18.c600fea5.1568726378.14abd986
Fidelity Investment OFX downloads are again working this morning, with no changes on my part.
DeleteHi Robert,
DeleteI love the development version's ability to import manually downloaded OFX files. Can you add an entry to SETUP for testing that folder (ie Just like we can test each online account and stock quotes we could have a way to run the importer routine)?
Here's my example Wealthfront for sites.dat. I used a localhost address to ensure that we wouldn't try to actually connect to a website. I included wealthfront in the URL to easily add a scrubber hook for siteURL. I pulled the fid and fiorg from my Wealthfront OFX file.
<site>
SiteName : WEALTHFRONT
AcctType : INVSTMT
fiorg : 8437
BrokerID :
fid : 17260
url : http://wealthfront.localhost #local downloads only
appid :
appver :
minInterval: 30
</site>
import scrubber feedback. Easy to setup and worked fine using Discover CC ofx download. Thank you again for all of your effort on this program. Was hoping we'd see Discover direct connect start working again, but sounds like the OAuth might kill it for us.
DeleteRobert, I haven't tried it out yet, but Google automatically places the downloaded file in Downloads. Couldn't your routine just work from that folder?
ReplyDeleteNever mind. Probably to prone to errors determining the path variance from os.path.curdir
ReplyDeleteI suggest turning on Chrome's setting "Ask where to save each file before downloading". Useful in general.
DeleteThanks redmon. I didn't know about that setting.
DeletePython 2 will be sunset (no updates, not even security updates) on January 1, 2020.
ReplyDeletehttps://www.python.org/doc/sunset-python-2/
I don't know if anyone has tried running pocketsense on python 3 yet.
It won't work in py3 without some code changes. Usually these are pretty easy, just changing the py2 modules to py3, and fixing up all the print statement to print functions. Out of curiosity, I spent some time yesterday walking through the basics of conversion. I didn't use 2to3 tool, just made hand edits (and regex replace) to fix up print statements and other things to make pylint happy. Still did not work at the end, due to some unicode problems on the pickle load. I was trying to re-use my working config files, and I may have to start a fresh testing environment.
DeleteI've started the upgrade a couple of times, starting with 2to3. I think there are issues with the decryption and some of the File IO. I have a version working without encryption, but was able to get the file i/o working yesterday. Just a couple of .encrypt() and .decrypt() additions were needed in ofx.py
DeleteRobert, is this something you want help with converting, I could work through it some more if so?
Thank you for the offer. Are you working w/ the latest (dev) code ?
DeleteMy main concern (and delay) w/ moving to V3 is that it will need to be a clean break, as we wouldn't want two independent code sets, and (probably) don't want to deal w/ having "dual support" in one code base either.
Robert, I hear you on the dual support. It looks like I have a working version from the beta, now for my purposes, and can share after finding a location to post.
DeleteFor Kevin, to get it to work I had to remove password encryption and delete and then regenerate the connect.key. The pickle.load seems to work when the encrypted data is removed.
files are here for now. https://github.com/bk-test/PocketSense2to3
Robert, I can
Robert the new script works great with Discover. Add to your notes, that you should delete the non-working accounts (Discover) in Setup at the same time you are revising sites.dat, so that getdata doesn't hiccup.
ReplyDeleteThanks for the revision!!
Dan (ameridan)
FIDELITY!!!
ReplyDeleteWhen I ran PocketSense this morning I got this in the XFR:
>HTML<>HEAD<
>TITLE<Access Denied>/TITLE<
>/HEAD<>BODY<
CL
>H1<Access Denied>/H1<
You don't have permission to access "http://ofx.fidelity.com/ftgw/OFX/clients/download" on this server.>P<
I went in to try again with "Test Account". Same result.
Dang. This will be bad, I fear.
CL
Just got off the phone with a Fidelity tech support rep. she said they were aware that quicken users were having a problem and they were working to resolve it. I told her that I used pocketsense and she said although it isn't supported by them, fixing quicken will probably fix pocketsense. Let's hope she's right.
DeleteThey have test platforms running quicken and she indicated that restarting their PC seemed to fix the issue. She was definitely knowledgeable and very competent.
I won't have access to my computer for a week and hope it'll be resolved when I get back.
Fidelity via Netbenefits did work for me just now.
DeleteHarold
Fidelity problem resolved on its own. I made no configuration changes, but just now I downloaded successfully.
DeleteCL
Same thing happened to me for Citicard this morning. Does Quicken have that much influence??
ReplyDeleteCiti worked fine for me. I just did a re-test to confirm.
Delete------------------------
SiteName : CitiCard
AcctType : CCSTMT #credit card
fiorg : Citigroup
fid : 24909
url : https://mobilesoa.citi.com/CitiOFXInterface
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval:
timeOffset :
I don't have access to my PC but I think your sites.dat entry is different from what I'm using. Will check when I return home in a week
DeleteGood to know I'm not the only person who's having problems with Citi cards again today.
ReplyDeleteCITI is still down for me too. I updated to the new url, but no go. I saw on an OFX site that you have to log-in to your CITI account and enable 3rd party access. I did that (it gives you a 10 minute timer) and still couldn't connect via pocketsense. I deleted the account and tried to add it again, but no luck.
ReplyDeleteThe ofxver:103 is important.
DeleteI think it may be important to relauch setup.py after making changes to sites.dat... Not sure.
When you launch setup.py, see that it signs on with this date or later:
PocketSense OFX Download Python Scripts, Ver: 24-Aug-2018
CL
The ofxVer:103 did it. It's working now. Thanks.
DeleteCITI is down for me also. (It is an AT&T account but serviced by CITI and had been working OK until yesterday, when I started getting the Access denied messages. Today I received a message, in the OFX download: "We've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement." That link took me to the previously mentioned page where you are allowed 10 minutes to set up the account. But it did not work. I've tried OFX 102, 103 and also the "new" URL: https://mobilesoa.citi.com/CitiOFXInterface. Suggestions appreciated.
ReplyDeleteTry this (BOTH these steps are necessary):
Delete1. Use a recent enough version of PocketSense that has the functionality to query the institution for your account list.
2. Execute the account list query within the 10 minute time window.
It's not sufficient to execute an account detail pull, you must execute the account list query within the 10 minute window.
Just so I understand, do I need to delete the accounts first and effectively start over again adding them?
DeleteYes -- or at least, I did. I ended up with a new connect key. Your mileage may vary.
DeleteTried that and it did not work.
DeleteUser name : xxxxxxxxxxxxxxxxx
Account password: xxxxxxxxxxxxxxxxxxxx
An error occurred requesting accounts from the site. Please check username and password.
I used the same name/password I use to sign into the website.
I didn't delete the account in Setup. I just switched the URL in my sites.dat, then started the 10 minute window and all was well again with my Citi card.
ReplyDeleteAfter trying various changes, I also eventually got Citi credit card downloads working, so I thought I would describe what worked for me. This is for a Costco Citi Visa card, and I am using the PocketSense August 24, 2018 scripts, so that is as current as it gets.
ReplyDeleteHere are the sites.dat parameters that are working for me:
AcctType : CCSTMT
fiorg : Citigroup
fid : 24909
url : https://mobilesoa.citi.com/CitiOFXInterface
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval :
timeOffset :
Before making any sites.dat changes I was receiving messages in the Citi OFX file that said: "Access Denied" and "You don't have permission to access https://www.accountonline.com/cards/svc/CitiOfxManager.do on this server." That is the url that was in sites.dat, and I was also using the default for ofxVer, which is 102.
After making the sites.dat changes, I did not need to use the Setup.py script, but I did need to use the Citi 10-minute process to relink the PocketSense scripts to my Citi account. The easiest way to do that is to log in to https://citi.com/datamanagement.
In looking at the downloaded OFX file, I saw that it said VERSION:102, so I tried removing ofxVer : 103 from sites.dat, but then I received this message in the OFX file: "We've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement."
I then put ofxVer : 103 back into sites.dat, and the downloads started working again, so that was good enough.
Thanks for posting... had exactly same problem with two Citi cards and both are working now with the sites.dat settings you used.
DeleteUnfortunately, still not working for me.
ReplyDeleteHere is my sites.dat
SiteName : AT&T
AcctType : CCSTMT #credit card
fiorg : Citigroup
fid : 24909
url : https://mobilesoa.citi.com/CitiOFXInterface
bankid :
ofxVer : 103
brokerid :
appid :
appver :
mininterval:
I just keep getting the message about going to the site to enable access, which I do.
Any thoughts?
How this worked for me was that after going to https://citi.com/datamanagement to enable access I then did a PocketSense Getdata.py transaction download during the 10-minute setup period. That transaction download worked normally, and after that the Citi transaction downloads continued to work normally.
DeleteOther people reported needing to use PocketSense Setup.py to delete and recreate their Citi account definition, which I tried before making the sites.dat url and ofxVer changes, but that did not work for me. Then, after making the sites.dat url and ofxVer changes, it was no longer necessary for me to use Setup.py to get the transaction downloads working.
A few months ago, I had a similar problem with American Express transaction downloads, and for that I did need to use PocketSense Setup.py to delete and recreate my American Express account definition. I also deleted and recreated my Citi account definition at that time because I was using an out of date account number that I was afraid would stop working. That was a pain because I ended up with duplicate transactions downloaded back to the beginning of 2019, so I did not particularly want that to happen again.
People who use Moneydance are reporting problems getting their AT&T Citi downloads working, so maybe there is something different about the AT&T Citi cards. Here is a link to a Moneydance forum where this is discussed:
https://infinitekind.tenderapp.com/discussions/online-banking/14145-citi-cards
Thanks for the link. Unfortunately, no solution posted yet for the AT&T/Citi people
ReplyDeleteYou could try posting here to see if anyone has any advice:
Deletehttps://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/comment-page-2/#comment-11241
Good luck.
Thanks. I've already been having a discussion with Cal and, unfortunately, his suggestions have not been successful
DeleteFinally started working again.
DeleteCould have been the complaints from the Moneydance people.
Could have been that I enabled access before setting up the account in PocketSense, instead of first setting it up, then enabling access, and then testing it.
Wow, lots of excitement since i was last able to log on to the blog.
ReplyDelete- My Discover downloads are still broken, but as was mentioned above by myself and @redmondman Discover downloads into Quicken ARE working. They go through a proxy-like setup though, rather than directly to ofx.discovercard.com
- Discover customer service was so far useless "contact your download service provider". They don't understand the service provider is ME (well, really Robert's scripts, but I don't wan to confuse them).
- Fidelity, Chase, BoA are not having trouble so far. I do not have Citi
Robert - I had written a pretty nice ingestion script for downloaded statements. I've been using it for months to processTarget and Transamerica, but haven't cleaned it up for broad distribution. I will send your way to see if it adds anything to what you've come up with.
What bank still works?
ReplyDeleteI have used Chase and Discover for decades with OFX and now with Discover dead I will start using just my chase visa but what bank options do I have?
I was ING Direct forever then then they were Capital one 360 then they stopped OFX and I changed to Discover bank, now they have stopped OFX so I need a new high interest saving bank that has OFX support? Suggestions please?
Heh. I've been a credit union member for decades, so I don't have a good answer. I've adapted through OFX outages, changes that required workarounds, and have finally resorted to custom code to work with my credit union (First Tech Federal).
DeleteWe're a dying breed; I just think it's going to keep getting harder.
" now they have stopped OFX so I need a new high interest saving bank that has OFX support? Suggestions please? "
ReplyDeleteAlly has high yield savings, but only has web OFX file download for savings. Still, it's better than most. No OFX for CDs.
I observe people posting about successful PocketSense access to Chase Bank, I think. But I cannot be sure because when they say Chase, without saying bank or credit card, who knows? Also, they will not meet your high yield savings need.
Your best bet would be a broker, and put you money into a premium MMF. They pay around 2% and you can get check writing. Several have good OFX via PocketSense.
There are times when you actually need a local bank. A local broker's office is not the same.
CL
In my experience, you can tell a lot about a bank (and most other service companies) by simply watching how they handle their clients.
DeleteFor example, American Express went through one of these conversions that affected OFX processing a while ago, and they were on top of it all the way through, telling you, in advance, exactly what you had to do -- and it worked exactly the way they said it would. In contrast, Capital One was completely brain-dead when they killed off OFX, and I closed all my accounts there because I viewed it as the "final straw".
Discover has always had pretty poor customer service, so I am not surprised by their left hand having no idea what their right hand is doing.
Harold
Nothing to add, but your comment about CapOne made me want to signal my agreement. Had been a customer since ~2005 and I finally switched out after the OFX and the requirement to have a mobile number (they were always trying two factor auth and I don't do that over text, nor do I want to give any company my mobile number).
DeleteI can add that I setup CITI cards today. Not sure if I had to delete them, but I deleted the accts in pocketsense, then started the online 10min timer and re-added them. They were doublecash (mine and SOs). I had 103 and the mobilesoa site in the sites.dat file. Hopefully it continues to work without issue.
Discover seems to still be dead in the water...waiting on that one until I see more comments from quicken or our community.
Any idea if any of these banks work with pocketsense?
ReplyDeleteAmerican Express Bank?
Barclays?
Marcus?
All 3 of those had hints from users posts that they might but want to hear from an actual pocketsense user to know for sure before applying.
I see etrade bank does not offer ofx (but brokerage does and I use that fine).
Barclays I believe is a no (at least credit cards). I haven't looked into it for years, but I don't think they do direct connect. I have to login and download my transactions manually every month.
DeleteFor anyone trying to get the Citi AT&T Universial credit card working, here's what I did just now.
ReplyDelete1. Changed sites.dat entry to:
<site>
SiteName : Universal_ATT
AcctType : CCSTMT
fiorg : Citigroup
fid : 24909
url : https://mobilesoa.citi.com/CitiOFXInterface
ofxVer : 103
</site>
2. Logged into Citi account center for AT&T card at https://www.citi.com/credit-cards/citi.action
3. Selected: Profile-->More Settings-->Manage Desktop Apps-->Add access, which started a 10 min window
4. Setup.py:
* Delete existing account.
* Add new account using above site.
* Select account number given in list. The account# (literally) looks like XXXXXXXXXXXX1234, w/ only 4 digits.
* Test == success
* Option 7: Test... success again.
When you send to Money, you'll need to link the new account# to the existing Money account.
- Robert
anybody was able to make it work with citi bank account? my citi credit cards work fine with new url - but not bank accounts
ReplyDeleteTo the best of my knowledge no one has gotten downloads to work for Citi savings or checking accounts. I have a checking account so if you figure out how to automatically download using PocketSense please post it here.
ReplyDeleteyes - before recent change i was able to login thru pocketsence and see accounts, but download always failed - now i even cannot login....
DeleteAgree, Ken.
ReplyDeleteEven the Citi bank OFX downloads from the website have become deficient compared to how it was a few months ago. There are FITIDs not staying the same for the same transactions downloaded later, and only one account can be downloaded at a time.
CL
Idea for scrubber element, to be called within scrubbers:
ReplyDeleteIf there is a <NAME> in a credit card transaction, consult a list of RegEx. With the first match, if any, replace the <NAME> field with the corresponding entry. Then create a <Memo> with the original Name content if there was no Memo already. If there was a <MEMO>, concatenate to the content up to a 255 character max.
Thus
THE HOME DEPOT #2036 JOHNSTOWN
THE HOME DEPOT #3917 SMITHVILLE
THE HOME DEPOT 130
THE HOME DEPOT 2036 JOHNSTOWN
THE HOME DEPOT 3917 SMITHVILLE
etc could all become "THE HOME DEPOT" or "HOME DEPOT", as you wish. The store info could be put into a <MEMO>
This would only affect new transactions, but you could consolidate in Money.
CL
Even worse is Amazon, where payee is unique for every transaction. I select my generic "Amazon.com" when accepting the downloaded transaction, but if the scrubber would do this, I'd like Cal's suggestion.
DeleteNxtTek,
DeleteHere's my Amazon.com trailing character scrubber:
Put this near the top of scrubber.py (after the "stat" line is fine):
def _scrubAmazon(ofx):
# 1. Scrub the trailing "*M12345678" from Amazon.com purchases on the Amazon Chase Card
p = re.compile(r'<NAME>.{0,12}\*[0-9A-z]{9}([<\n])',re.IGNORECASE)
if p.search(ofx):
ofx = p.sub(lambda r: '<NAME>Amazon.com' + r.group(1), ofx)
scrubPrint("Scrubber: Amazon.com order numbers removed for aliasing.")
return ofx
Put this just above the DISCOVERCARD and TROWEPRICE sections of the scrub routine in scrubber.py:
if 'CHASE' in siteURL: ofx= _scrubAmazon(ofx)
I use this on my Chase Amazon.com card. You can adjust the location of the scrub if you use multiple cards for your Amazon purchases. The REGEX finds a name with 0-12 characters, a *, and then exactly 9 letters/numbers. This matches Amazon.com, AMZN Marketplace, etc. I haven't had a false positive yet. You could make it more specific to prevent that from occurring.
Andrew , NxtTek, and all:
ReplyDeleteI think for a function used by non-programmer people, that the data for a function might be entered something like this:
PS_payee_meld_table=(
("Amazon.com","Amazon"), #if it starts with Amazon.com, change to Amazon.
("AMZN Marketplace","Amazon"), #similar
("THE HOME DEPOT","HOME DEPOT") #Consolidate Home Depot payees.
)
The ^ at the start of the string could be done by the function rather than the user. Where the desired action is to switch if it contains, then enter the something like ".*wendys" and so on.
Or the data could be read in from a csv file rather than a Python file, so the user could prepare the list with a spreadsheet if desired. MSMoneyQuotes does it that way for a lookup table. Perhaps pass the name of the spreadsheet as a parameter. My concern would be if a payee contains a comma.
I would propose that the function break out the Name field contents, and each name be compared to the lookup tuple until one is found or nothing is found. In this way, no bad RegEx could do bad things other than cause a mismatch. Andrew, you have the knowledge to get a RegEx right, and the use that in a sub(). RegEx are darned easy to get wrong, so let any mistake only cause no action or a wrong payee entered. Limited consequence.
Cal Learner
"A lot of programmers see a problem and think, "I know! I'll use a regular expression!" Now they have two problems."
DeleteI've been using PocketSense for years and I just ran into a new problem. GetData.py does not work (does not request password and says, "No accounts have been configured. Run SETUP.PY to add accounts.") ofx_config.cfg is present, populated, and unchanged from several months ago. Running Setup.py does not request password and shows no accounts. sys.version_info(major=2, minor=7, micro=8, releaselevel='final', serial=0). Any suggestions?
ReplyDeleteThe ofx_config.cfg file contains the setup info for your accounts.
ReplyDeleteI occationally back mine up. If your ofx_config.cfg is very small, maybe that file got lost.
CL
Paul,... I re-read. Sorry. I don't know why setup.py does not read your ofx_config.cfg.
ReplyDeleteCL
TD Bank - I was able to get the TD Bank download to work, couple of issues that are dealt with to setup
ReplyDeleteFirst - the settings
SiteName : TD Bank
AcctType : BASTMT #bank
fiorg : CommerceBank
fid : 1002
url : https://ofx.tdbank.com/eftxweb/td.ofx
bankid : Required – enter routing number
ofxVer : 103
Second - run setup
After entering username and password you will get notified of an error, now just leave setup open and running
Third - after about 1 minute log into the TD website for your bank account
Go To Account Options-Account Services
In the manage linked services box, under financial tools, click Manage Settings
Accept/Approve Quicken 2015
If necessary, log out and log back in to try again and get approve option
Fourth - Go back to setup and say yes to configure manually
Enter account number
Choose account type
At this point the download should work (I cant remember if you have to say yes to test, yes to send to money or yes to both, but it should work)
Hope I remembered the process correctly
I thought I'd gotten TD Bank to work following Michael's method, but it isn't. I'm getting through the first check at TD Bank but then there's some kind of hiccup. If anyone can figure out what's causing the "Duplicate Request" error I'd greatly appreciate it.
ReplyDeleteOFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
{OFX}
{SIGNONMSGSRSV1}
{SONRS}
{STATUS}
{CODE}0
{SEVERITY}INFO
{MESSAGE}SUCCESS
{/STATUS}
{DTSERVER}20191028175838.000[0:GMT]
{LANGUAGE}ENG
{DTPROFUP}20171101040000.000[0:GMT]
{DTACCTUP}20191028175838.000[0:GMT]
{FI}
{ORG}CommerceBank
{FID}1002
{/FI}
{/SONRS}
{/SIGNONMSGSRSV1}
{BANKMSGSRSV1}
{STMTTRNRS}
{TRNUID}43adf3c9-9426-417a-9b35-69849515fc93
{STATUS}
{CODE}2019
{SEVERITY}ERROR
{MESSAGE}Duplicate Request
{/STATUS}
{/STMTTRNRS}
{/BANKMSGSRSV1}
{/OFX}
Ken, from this site http://libofx.sourceforge.net/html/fx-0_86_85_2lib_2ofx__error__msg_8hh-source.html
ReplyDeleteDuplicate Request = "Duplicate request", "A request with this has already been received and processed."
Maybe try closing setup and retrying, but skipping the part about TD website as that is now activated. In setup after entering username/password and you get the error, say yes to configure manually and enter account number, etc
If necessary you can delete the approval from the TD website and start over. Not sure exactly how I got it to work
The error message got altered, here it is again without the problematic brackets
ReplyDelete"Duplicate request", "A request with this TRNUID has already been received and processed."
Are you using the most recent version of the scripts? The TRNUID is unique for each request. However, I don't recall if a static value may have been used in old versions (?).
DeleteA quick question for the forum - not really a Pocketsense query but a MSMoney one.One of my financial accounts only offers downloads as a Quicken file (.QFX or .QBO). Until today I have been able to download a file in QFX format,change the extension to .OFX and then Moneyhandler would process the file happily. As of this morning I am getting a message from Moneyhandler saying the file is either invalid or contains corrupt data. I'm assuming Quicken have made some sort of change to their transaction formats and wonder if anyone has found a way to continue to import their files into MSMoney?
ReplyDeleteThank You
Try the latest dev/test version. See this post above
DeleteI download QFX files from Discover Card.
DeleteI don't bother to rename them to OFX though. I just change the 'Files of Type' drop-down to 'All files (*.*)' in the import window.
The QFX file imports as expected.
Robert's latest dev/test version does a great job of importing the files automatically, if saved into the import folder. It takes care of the extension for you, as well, so just leave it as .qfx
DeleteWorks great! Thank You!
ReplyDelete> Discover seems to still be dead in the water...waiting on that one until I see more comments from quicken or our community.
ReplyDeleteThe Http Header Host needs to have a port number. So try using the following the setting in the sites.dat file
url : https://ofx.discovercard.com:443
That worked for me this morning. YES!! I'll add this to my settings article @
Deletehttps://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
hleofxquotes and NxtTek: Success is good! But mysterious in that 443 is the default port for HTTPS. I think there must be a quirk somewhere, and this works around it.
ReplyDeleteCal Learner.
I didn't try Discover this morning with the standard sites.dat entry. Perhaps that works too? I just tried the suggested change and "discover"ed that it works ;)
DeleteNxtTek: if you tried, and it did not work, as an experiment, you might try url : https://ofx.discovercard.com?
DeleteWITH the question mark. The theory would be that maybe the original URL got cached. The ? would make the string different, and would not use the cached version.
Before posting this, I tried adding a question mark at the end of the Vanguard URL; that worked in a test.
CL
The work-around detail:
ReplyDeleteThe request http header for "Host:" must look like this
Host: ofx.discovercard.com:443
Using a URL with the port number such as https://ofx.discovercard.com:443, will make that so.
Thank you for posting. I confirmed that it works for both Discover Card and Bank. For anyone curious, the https connection uses port 443 (as Cal mentioned), but explicitly putting it in the URL means that it also gets included in the HOST header in the http/POST request. Go figure as to "why" it changed...
DeleteHas anyone used the .csv and .json to .ofx converter from: https://github.com/kedder/ofxstatement ?
ReplyDeleteI opened an account at Simple Bank and they only provide downloads in .csv and .json formats so I went looking for a converter. Is anyone using a converter that's easy to use and straightforward? The one I found at github requires Python 3 and I don't know if there'll be a conflict if I have both Python 2 installed for PocketSense and Python 3 for this app.
As usual, thanks for your help.
Ken: https://www.activestate.com/products/python/python-3/ says " Since you can safely install both ActivePython 2.7 and ActivePython 3 side-by-side on your computer, you can experiment with Python 3 while still using Python 2 for your current code."
DeleteCL
Hi Ken,
DeleteProvided that you don't mind using qif instead of ofx, take a look at xl2qif.
It's an add-in for Excel. As the name implies, it converts csv to qif.
It works well for Bank and Credit Card transactions. It's free to use and easy to set up.
http://xl2qif.chez-alice.fr/xl2qif_en.php#download
-Kevin N.
@Kevin Thank you! Already downloaded the Add-in and figured out exactly how to use it. It's pretty spiffy and will make it much easier to keep Money up to date for those institutions that don't provide a QFX or OFX download capability. Really useful.
ReplyDelete@Cal Thanks for the Python info. The Add-in that Kevin suggested will probably do the trick. If I have some free time over the holidays I'll try and get the Python OFX converter to work and see if it imports more robust detail. But for a quick and dirty, the Add-in definitely makes things easy.
ReplyDeleteI cannot believe discover works again! I changed everything to Amex after 20+ years of downloading discover transactions. And then just this morning I deleted all my discover bank accounts (6 of them)... I was unable to find an alternative bank that worked with pocketsense that was high interest rate, I guess now I don't have to! I actually came here to see if anyone had posted if anyone had any other banks. BTW discover told me...
ReplyDelete"Due to a recent update to our website, we can only accept OFX connections from Quicken, QuickBooks, and eMoney. Other financial software will no longer be able to connect. I apologize for any inconvenience this causes."
Glad they are now wrong.
Ok I set everything back up and did find a problem with discover card's ofx server. They must have some sort of control on it based on IP or something to prevent multiple hits in a row. When I set all the accounts up and test them (with a few minutes in between) they all worked fine. I then used the script to download them all and the first 3 (of my 7) worked. The final 4 got access denied. I then waited a minute and manually checked one of the ones that was denied through the test in setup and it worked fine again. I then ran the entire script again and only the first account worked the other 6 came back with access denied. Finally I ran the whole thing again right after and all 7 accounts were denied.
ReplyDeleteI have played a little with this but can someone else test making a request a few times in a row and see if they temporary ban you like it appears is what is happening to me?
I have 2 Discover Card accounts.
DeleteGetdata.py successfully connects to both of them without incident.
Perhaps there is a delay-timer that can be set within the scripts to prolong the time between connection attempts.
-Kevin N.
Kevin can you do me a favor. Go into the setup and test the account several times in a row and see if it eventually gives you the "You don't have permission to access" error.
DeleteYes I think the delay might work great, I just want to verify that is in fact what is going on. Thanks
Hi Anonymous,
DeleteI ran through 'Test Account' several times for 1 of the DC accounts without incident. Then again alternating between the 2 accounts. Again, without incident.
-Kevin N.
Thanks for checking. I just tried it and it did 5 of the 7 before it would not allow it again. Maybe I will add some fake account or something between them in the script that will need to timeout in order to cause some delay because it does certainly appear to lock me out after hitting it enough times in a row.
DeleteIs there an easy way to change the order that it checks the sites in? Trying to mix my discover bank accounts in with the other accounts to hopefully get past this blocking issue. If not an easy way to add a delay? thanks
ReplyDeleteSure there is. Create A DISCOVER & Z DISCOVER (duplicates of each other) in sites.dat, as an example. Then add some accounts using each.
DeleteThanks, so I can add more discovers to sites.dat however when I check them it will always cycle through them in the order I added them in setup (or the order that I see if I list accounts in setup), not in the order the banks are in sites.dat. I have 14 accounts that I have pocketsense check (and then investments) and it does those accounts in the order I add them. I was trying to avoid just going and and clearing them out and readding in a different order. Wondering if there was a cfg file or something I could just edit and change the order.
DeleteIf you sort your xfr folder by date modified, I think you'll see that the download order is alphabetical. Same with the combined file, if I'm not mistaken, which is why I made the recommendation of "A DISCOVER & Z DISCOVER".
ReplyDeleteThanks again, but mine does seem to be doing it by order they are listed in the setup. My xfr folder goes etrade, Vanguard, Fidelity... and so on the exact same order as it downloads them and shows them in the list.
DeleteI was able just to add a few bad login etrade attempts between the discover attempts and that slowed it down just enough that it no longer gave me a security issues and downloaded all 7 accounts. Thanks
DeleteSo... you only need a second or so between attempts? I need to get the current test version posted, and could add a delay option, prob at the Site level? The idea there would be to have a user-defined delay to insert before attempts to any account defined using that site entry. Would that help?
DeleteAs for account order, it's not supported. Account entries are "pickled" into a linear/flat file named ofx_config.cfg as they're created. While much of it is encrypted, there's a distinct storage pattern, and "blocks" could be moved around to change order. It would be a hack, so if done wrong, would break it.
I could probably (easily) add a simple sort option, if needed. Something like "sort alphabetically by site name up or down". Useful?
The fake account trick between my discover accounts is working great. Sure a delay I could set would also work and look nicer but in the end it does work and if I'm the only one with the problem I would not worry about it. Thank you again for this software! When discover was out I looked at the alternatives again and NONE could do anywhere near what good old MS money can do (along with pocketsense).
DeleteRobert,
ReplyDeleteI think the delay option by site would be great, and would probably solve this issue very nicely -- and would be much appreciated.
Harold
I use a Windows scheduler to run GetData.py early mornings Tuesday through Saturday. The reason is to not get quotes imported into Money investment accounts for Saturday or Sunday. Quotes are a big factor in how big a *.mny file grows.
ReplyDeleteIf there were a command line switch to let me restrict or specify accounts to be downloaded, I could invoke with the switch to get the credit cards over the weekend. I don't even know if they get posted over the weekend. If this were a bigger deal, I could use two different PocketSense folders, and use one or the other depending on which accounts I want for that day. Not a big enough thing for that for me.
CL
Has anyone else experienced a drastic slow down in quotes for mutual funds and/or stocks within the past few days? My quotes are taking about one minute each to process and I obtain +/- 50. First noted on Thursday, Nov. 14th.
ReplyDeleteI've had a few slow days, but it seems to fix itself a day or two later.
ReplyDelete