This post is for the purpose of testing the latest DEV version and tracking comments. It includes a few updates, including an attempt at adding support for USAA. It adds a new dependency named Requests that may not be included in base python distributions.
Unfortunately, I can't give exact instructions for installing python dependencies, since it varies by distribution and install options. Some already include it, such as ActiveState and Anaconda. Generally, I recommend the ActiveState Python 2.7 distribution for general purpose use, which includes Requests by default, plus a number of other popular community-developed packages.
Generally, the following command will install the package for a lightweight/basic python installation. From a command prompt (1):
pip install requests
The scripts require Python 2.7.x, where x should be above 11. I'm currently using 2.7.15. The following link will be active for the duration of this post, and any reference I give to the current "dev version" means the following:
Download --> DEV Version using Requests
- Robert
(1) If the python installation isn't in your system cmd path, then you will need to open the command prompt in the folder where it's installed.
Change Log:
- Changed to using the Requests library rather than httplib.
- Added the following sites.dat global parameters:
- promptStart: prompt/pause to continue when starting getData
- promptEnd : prompt/pause to continue when getData is finished
- Added the following site-specific parameters:
- dtacctup: site over-ride for DTACCTUP field in ofx request
- useragent: site over-ride for USERAGENT field in ofx request
- clientuid: site-specific (user provided) clientUID value in ofx request. This option requires ofxVer=103 and should only be used for good reason (e.g., USAA requires it).
- Scrub routines:
- Incorporated scrubber.py modifications provided by Carlos N.
- Added "remove null TRNTYPE fields" to _scrubGeneral().
- Added support for custom scrub modules (scrublets). Any scrub_*.py file found in the same folder as the base files (setup.py, getdata.py, etc) will be loaded and run for each statement. The scrub module must contain a function named scrub([string]ofx, [string]siteURL, [string]acctType), and return a valid ofx message.
- Routines for Discover and TRowePrice were moved to scrub_discover.py and scrub_troweprice.py, and are included in the base distribution (zip) file. These, along with scrub_example.template give examples of custom scrub modules.
Kicking off w/ a clean slate, w/ minor updates from the latest comments.
ReplyDeleteWith the Sunsetting on OFXHome, where else can I find/search for FI info on Mid Penn Bank of Pa?
DeleteIn the process of changing and seemed to have spent an enormous amount of time looking for this info. It is not in the OFX download from the institution; I need FI Org & FI URL.
Any help would be appreciated as I've used PocketSense for years.
Cheers
Jeff: I find no info for you, and I expect they have no accessible server.
Deletehttps://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/ has PocketSense descriptor info for known FIs.
Also, GnuCash can be useful for finding info.
Cal Learner
All of my known sources are linked to here:
Deletehttps://microsoftmoneyoffline.wordpress.com/look-up-your-ofx-settings/
This comment has been removed by the author.
DeleteBeen there, done that, but thanks Cal & NxtTek. Over 30 bank locations & since the 90's that I'm aware of throughout PA, I'd have thought this info would be somewhere but I've been fruitless. At least they support OFX downlds. Wonder if one of their IT people would divulge....
DeleteTheir web-portal does have an additional rotating security question, but perhaps no matter as another bank I use thru PocketSense is fine; that bank uses a different url in sites.dat than for web entry (https://ofx.netteller.com)
Cheers, Jeff
Last DEV version using Requests was working with exceptions as noted in the last thread. This version not working -- selecting either setup.py or getdata.py --> the window flashes open and then immediately closes.
ReplyDeleteFixed.
DeleteOK This version is now working for USAA (and probably AT&T and Chase), but still FAILING for Fidelity with `Bad Request` being the response from Fidelity, as before.
ReplyDeleteNOTE:
I commented out the old USAA sites from site.dat.
When I added back my USAA accounts using the new site, the old accounts still showed up on the list with a notation that there was no entry in sites.dat
At that time, getdata.py would NOT create the combined.ofx file, nor send OFX data to MS Money
After deleting those accounts using setup.py, getdata.py worked (for USAA) as it should
To verify (see below)... you added the "userAgent: none" parameter to your site entry for Fidelity before testing?
DeleteI think I'm missing a thing.
DeleteThe Java app works for USAA with the provided GUID as clientUid, the eight-character not-a-username as user.id, and the six digit number as my password.
If I do the equivalent with the new version of PocketSense, and the following as the Sites.dat entry:
SiteName : USAA
AcctType : BASTMT
fiorg : USAA Federal Savings Bank
fid : 67811
url : https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
bankid : 314074269
brokerid : 5874
appid : QMOFX
appver : 2300
mininterval:
timeOffset :
ofxVer : 103
dtacctup : 19900101
userAgent : InetClntApp/3.0
clientUID : (my GUID)
then I get the Incapsula response rather than my transactions.
Did I skip a step somewhere in there?
TESTS
ReplyDelete1. Add the following to the site entry for Fidelity:
userAgent: none
2. Not loading? ?... try adding a delay (seconds) to each of the USAA site entries, since you have multiple entries using the same params it may be getting blocked at the gateway.
delay: 2
I added userAgent: none to previously working Fidelity sites.dat
Delete```
SiteName : FIDELITY
AcctType : INVSTMT #investment
fiorg : fidelity.com
fid : 7776
url : https://ofx.fidelity.com:443/ftgw/OFX/clients/download
bankid :
brokerid : fidelity.com
appid :
appver :
userAgent : none
mininterval:
```
Fidelity still returning 'Bad Request'. And still OK using the last Release version
-------------------
OFX's not downloading to MS Money was fixed by deleting the non-functional (old) accounts USAA accounts using setup.py
TEST: Remove userAgent: field, and remove ":443" from the url (i.e., the ssl port).
DeleteStill getting `Bad Request`:
ReplyDelete```
SiteName : FIDELITY
AcctType : INVSTMT #investment
fiorg : fidelity.com
fid : 7776
url : https://ofx.fidelity.com/ftgw/OFX/clients/download
bankid :
brokerid : fidelity.com
appid :
appver :
mininterval:
```
Ron - can you turn on Debug and print sanitized versions of OFX request and response? Fidelity works for me.
ReplyDeleteAre you getting success with Fidelity with Robert's version? Or with your own edited version?
DeleteHere's a link to two debug files generated when using the Add account option from setup.py -- one is a success using the last Release version; the other a failure using the latest DEVwithRequests version.
https://1drv.ms/u/s!Aml6o6HIc9uOrHunw5IRPl243c70?e=4dU39g
Both use the secure port and neither uses the userAgent line as that was successful with the Release version.
Thanks.
Ron
By the way, the DEV version also failed when removing the secure port 443 and adding `userAgent: none`
DeleteUpdate posted & ready to test.
DeleteRon - Below you'll find the OFX string and http headers that I'm sending over to Fidelity using my development version (I'm using the requests library just like Robert is now using). Based on my results, you don't need to include the 443 port number at the end of the URL. The remaining deltas would appear to be the date format (Robert is using UTC, I'm using GMT), the DTACCTUP date string, and the User-Agent. The requests library is including User-Agent regardless of whether it is populated in the user code. I'm sending over PocketSense as my User Agent. I would suggest changing the URL, populating the User-Agent field, and modifying the DTACCTUP field and trying a request using that data. Post your results of that testing.
DeleteurlHost : ofx.fidelity.com
urlSelector: /ftgw/OFX/clients/download
OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
20210225085655.235[-6:CST]
xxxxxxxxx
xxxxxxxxx
ENG
fidelity.com
7776
QWIN
2600
4016bef0-fd0e-460f-a2d0-9a2f11e89498
19700101000000
Here's the header content that's being passed via the requests library:
{'Content-Length': '516', 'Accept-Encoding': 'gzip,deflate', 'Host': 'ofx.fidelity.com', 'Accept': '*/*', 'User-Agent': 'PocketSense', 'Connection': 'Keep-Alive', 'Content-Type': 'application/x-ofx'}
Here's the URL:
'https://ofx.fidelity.com/ftgw/OFX/clients/download'
Welp, that worked great. Should have removed the OFX tags. Hopefully you can piece together the field names based on the values.
DeleteRobert: New DEV seems to still work with USAA on a test; still Fails with Fido
DeleteJohn & Robert: I changed my Fido sites.dat as John suggested:
```
site>
SiteName : FIDELITY
AcctType : INVSTMT #investment
fiorg : fidelity.com
fid : 7776
url : https://ofx.fidelity.com/ftgw/OFX/clients/download
bankid :
brokerid : fidelity.com
appid : QWIN
appver : 2600
dtaccctup : 19900101
userAgent : PocketSense
mininterval:
```
Try changing DTACCTUP to the January 1, 1970 date string.
DeleteAlso verify your headers are similar to what I posted.
DeleteHeaders won't match above, but should match the httplib (release) version of the scripts. Just fyi...
DeleteAlso... the following site entry would match the "release" version exactly:
DeleteDTACCTUP: 19700101000000
Robert: Changed dtacctup:19700101000000 in Fido sites.dat entry and test still fails with Fidelity.
DeleteRon: If you try to add the account from scratch (Setup --> Add Account), does it connect?
DeleteRobert: No, it does not connect
DeleteWondering if the DTCLIENT data is the issue. I'm using GMT (i.e. -6:CST) format and everything works correctly.
DeleteJohn: In the penultimate DEV version, the fido dtclient IS being sent as GMT: `20210225014330[+0:UTC]` was in the debug logs.
DeleteWorth checking. Update posted, but w/ *no* GMT suffix, since we've never used it in the past.
DeleteThat last fix seems to have done the trick. Both Fidelity and USAA connected and downloaded to MS Money without difficulty. Hooray!!
DeleteAnd Robert, as an aside, in the sites.template file in this distribution, in the empty site.template, dtaccctup is misspelled. should be dtacctup (2 c's instead of 3)
Here is my FIDO sites.dat for the record:
```
SiteName : FIDELITY
AcctType : INVSTMT #investment
fiorg : fidelity.com
fid : 7776
url : https://ofx.fidelity.com/ftgw/OFX/clients/download
bankid :
brokerid : fidelity.com
appid : QWIN
appver : 2600
dtacctup : 19700101000000
userAgent : PocketSense
mininterval:
```
Progress! If you don't mind, remove the dtacctup and userAgent values from the site record, and test. Curious if those matter...
DeleteMinor update posted. It would be useful if someone can test/verify that "add account" gives a list of accounts for USAA, or any other for that matter. No need to actually add, just +c out after the list shows.
DeleteThat should have been: <Ctrl>+c to break
DeleteRobert. Using the most recent update, and removing dtacctup and useragent from FIDO sites.dat entry: All seems to be working still. And using "add account" will produce a list of accounts for USAA (and also for FIDO. Valid OFX files are now downloaded for both!!!! (as well as my other FI's (AT&T credit card and CHASE credit card)
DeleteRon on February 25, 2021 said "Here is my FIDO sites.dat for the record:"
DeleteRon, that will probably lead to the dates on your price history being a day or so off. So you will want to look into the time offset feature.
I use my own Fidelity scrubber with a more complex system, but I think the timeOffset will make a big improvemnt. I don't have a timeOffset value to suggest right now.
Cal Learner
I think that for some reason, my posts were not making it to the previous collection of posts. I just wanted to ask if there was any hope of getting Discover (card and bank) to work with Pocketsense... could they be doing the same type of thing that USAA is doing?
ReplyDeleteHarold
Your question did post, but you have to click the "load more" link at the (apparent) end of comments, since it only shows the first ~200. See reply here.
DeleteWell... the link above just opens the comments. "Load more" at the end, and then circa Feb 20/21 comments.
DeleteI can confirm that the PocketSense version as of 25 Feb 2021 works for me for downloading USAA transactions. Thanks Robert!
ReplyDeleteI am able to use the dev version to connect to USAA in setup and get the list of accounts. But when I try to download statements under the test menu or the getdata.py program, it fails immediately.
ReplyDeleteThe error in the xfr folder is:
Error encountered. See details below:
java.io.IOException: Server returned HTTP response code: 400 for URL: https://usint.pz.finance-api.services/eftx/usaa/access.ofx
my sites.dat is:
SiteName: USAA
url: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
fiorg: USAA Federal Savings Bank
fid: 67811
bankid: 314074269
AcctType : BASTMT #bank
appid: QMOFX
appver: 2300
ofxVer: 103
dtacctup: 19900101
userAgent: InetClntApp/3.0
clientUID:
any suggestions?
The clientUID I used was the number in the USAA Quicken Authorization page URL: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
DeleteIn my notes, it states: Your_ClientId_From_The_Allow_Page, which may be different from the Authorization page?
DeleteIs your python version 2.7+ and does it include `requests`? From a command prompt, `python -V` for the version. `pip install requests` for the module. If it is already installed, it will not reinstall and you'll get appropriate messages.
DeleteI got this working.
DeleteLike NxtTek said, there is a difference in Your_ClientId_From_The_Allow_Page and the next screen where you get the AccessID and AccessPIN.
The ClientUID from www.usaa.com/accessid was the one that worked for me. I also noticed that this ID doesn't seem to change but the one from the next screen changes every time.
The last step was to delete the account in setup.py and set it up again once I had the correct ClientUID in sites.dat
Thanks Robert for getting this working, and thank you for the hints Ron and NxtTek!
I can also confirm that I have my USAA accounts working with this version (2 checking, 2 savings, and 1 credit card). The sites.dat entry is the client id from the URL on the allow/deny page. For the credit card, I copied/pasted the sites.dat entry, changed the AcctType to CCSTMT, and modified both SiteName entries for clarity (USAA Banking, USAA Credit Card).
ReplyDeleteSome other modifications that I make to my scripts each time:
Getdata.py
1) I add "#!python2" to the top of Getdata.py (and Setup.py). If you also have Python 3 installed, this will ensure that your system uses Python 2 to run the script on double-click.
2) I remove the prompt to download files (at the "doit" line, comment it out and below it put "doit = 'Y').
3) I like the opportunity to review the log before it disappears, so at the very bottom I put this to require input to close the window:
else:
raw_input("ALL GOOD! Press to continue...")
Scrubber.py
1) I've mentioned this before, but I add a scrubber for my Chase Amazon credit card to remove the random characters that Amazon puts in the merchant name for each order. This allows MS Money to always recognize the merchant as "Amazon.com".
===============
def _scrubAmazon(ofx):
# 1. Scrub the trailing "*M12345678" from Amazon.com purchases on the Amazon Chase Card
p = re.compile(r'(Amazon\.com|AMZN Mktp US|Kindle Svcs|Audible|Prime Video)\*[0-9A-z]{9}',re.IGNORECASE) # e.g. Amazon.com*2T5LI9LA2 or Amazon.com*MK4J845L0
if p.search(ofx):
ofx = p.sub('Amazon.com',ofx)
scrubPrint("Scrubber: Amazon.com order numbers removed for aliasing.")
return ofx
===============
Add this in the "scrub" routine, somewhere near the Discover and TRowePrice lines:
===============
if 'CHASE' in siteURL: ofx= _scrubAmazon(ofx)
===============
Thanks Andrew. I can add options for most of those, except the scrub routine. The scrubber module is intended mostly for correcting issues that break the ofx import in Money vs what a server sends, although admittedly some of these are site-specific.
DeleteI've thought about adding support for custom scrub modules, but haven't focused on it. This update will be more extensive than most recent changes, so worth considering. Something along the lines of "any file named scrub_*.py will be loaded and processed for a single call to _scrub(ofx, siteURL, siteType)." The module file would contain an entry function named something like _scrub(ofx, siteURL=None, siteType=None), and return ofx. Would this be useful? I think it would be pretty easy to add.
Hi Robert,
DeleteI think that anyone writing a scrub module would be familiar enough with Python to handle either scenario.
I think that the driving question for reorganizing the code is, does it make the code easier to read, understand, and update. For that question, I think you would be fine with either solution.
So, follow your heart. :)
-Andrew
Robert said "I've thought about adding support for custom scrub modules, but haven't focused on it. This update will be more extensive than most recent changes, so worth considering. "
DeleteI think that is a very good idea. I had thought of the name "scrublet. This could be useful for things that pop up where a provider starts intermittently turning the upper characters of the account number into X's. Who knows what some provider will come up with next.
I routinely run code to make the OFX easier to read by adding newlines in front of < that are not followed by / characters, but only when they do not already exist. That would be an example of one that would not need to be provider-specific, but could be global.
Cal Learner
Robert,
DeleteI have already implemented custom scrubber modules in my personal copy of PocketSense.
In sites.dat, you would add an optional "SCRUBBER:" directive to tell PS what module to python module to load at runtime. The module must have a public function def _scrubber(ofx, accType='CCSTMT'): that returns the scrubbed ofx, plus any needed code to accomplish this. scrubber.py needs about 30 lines including comments to dynamically load, run some basic sanity checks, and execute the scrubber.
I have written three "custom" scrubbers: scrubDISCOVER.py which offloads the old DISCOVER scrubber to a loadable module, scrubTARGET.py to support Target Red Card, and scrubTROWEPRICE.py for T. Rowe Price.
I have made numerous changes in my copy, including a fix for the Wells Fargo headers. Let me know if you would like to see the changes.
I have been running these changes since last August on my Money files.
Andy Dingwall
That sounds very close, except I was thinking it would pass the pertinent site parameters and leave when/how to process in the "scrublet", as Cal called it (which I like!). All available scrublets would be called for each ofx response, but would only apply to whatever matching url/acctType it deemed proper. If it were being applied to all sites/acctTypes, then I guess that makes it a proper candidate for scrubber.py.
Delete"If it were being applied to all sites/acctTypes, then I guess that makes it a proper candidate for scrubber.py."
DeleteMaybe, but for example, imagine something like a pretty-print operation. Somebody might want that operation applied to all FIs. People might differ on what fits their thought about what makes good formatting for readability of the OFX files.
Cal Learner
@Cal: I thought the same about the time I hit "post". The amazon cleanup mentioned above is another. Some may like it, others not.
DeleteExample of a potential scrublet
DeleteI have a function I call in my Fidelity scrubber "remove_unwanted_OFX_transactions(ofx)". It removes transactions of a specified type that contain a specified string. The code is written fairly generically, but the specifications for type and string are hard coded into a tuple definition. "A tuple is a collection of objects which is ordered and immutable."
My specific uses are to to remove all BUYMF and SELLMF for my sweep money market fund, and to remove TRANSFER transacitions that contain a specified string. So the first entry within the tuple is
("BUYMF","316067107"), #delete BUYMF transactions that contain 316067107 (a CUSIP)
I have three entries in my tuple, but there could be more. Because the tuple vaues are hard coded, this would be appropriate for a personal scublet. If upgrading to a later version of PocketSense, this could continue to operate without modifying the new release code.
Cal Learner.
Thanks for the Amazon scrub Andrew. I added it as section 3 in def _scrubGeneral(ofx) of my scrubber.py instead, since my Amazon purchases are usually on my Discover card.
ReplyDeleteRobert, is there a download available?
ReplyDeletejmf
JMF: If you are looking for the latest "released" version, see https://sites.google.com/site/pocketsense/home/msmoneyfixp1/p2
Deletestep 1b. So that dates back to January 2020.
The "DEV Version using Requests" link above gets you to an evolving new version that is still under development. So that is for the cutting edge folks contributing to trying the code being worked on.
Cal Learner
This does not work for USAA Checking. Python reports that my account number replaced by sitename, and error messages: "invalid ofx statement detected," then gives me a link to review. The link begins with ".\xfr\" not sure what to do with it. Please advice, thank you.
ReplyDeletejmf
SiteName : USAA
AcctType : BASTMT #bank
fiorg : USAA
fid : 24591
url : https://service2.usaa.com/ofx/OFXServlet
bankid : 314074269
brokerid :
appid : QWIN
appver : 2300
mininterval:
timeOffset :
That's the old ofx site. The appid is also old. See some of prior discussion for what the sites file should look like going forward.
DeleteI guess I need to be spoon-fed. Please provide a sites.template for USAA Checking That works with the new ID and Pin, thanks.
Deletejmf
See https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
DeleteOkay, its not working. I used the below site entry and put my ID from the website in the 2nd last line (not my pin). I deleted the old account and added it back using my user name and password (same as required to assess my online USAA account page). I'm also using the newest Rob files from the 2021 DEV site files. What am I doing (not doing) wrong? Thanks
Deletejmf
SiteName : USAA BANKING
AcctType : BASTMT #bank
fiorg : USAA Federal Savings Bank
fid : 67811
url : https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
bankid : 314074269
brokerid :
appid : QMOFX
appver : 2300
ofxVer: 103
dtacctup: 19900101
userAgent: InetClntApp/3.0
clientUID: Your_ClientId_From_The_Allow_Page
mininterval:
I don't use USAA, but my understanding is that you use the access id and PIN provided on the signup page as your username and password, respectively.
DeleteOkay, so what do you put here: "clientUID: Your_ClientId_From_The_Allow_Page"
Deletejmf
Maybe a USAA user can provide better info, but I believe it's in the signup URL (browser url when you're signing up). It will have the following form:
Delete1955A543-B071-455E-A31E-73CC7C493D68
I'm not sure if it's the same for all ofx users, or if each person gets their own (?).
1) Sign in to USAA.com
Delete2) Go to https://www.usaa.com/accessid
3) You will be at a page to authorize Quicken to access your USAA information (it's not really Quicken, just enabled OFX server access). From the URL on this page, get the client_id field. It has the format that Rober described above. You need everything between the leading "=" and the trailing "&". This number does not change for your account.
4) Click "Allow" to enable access.
5) On this page, copy your user ID and PIN. These do not change for your account.
Put the client_id in the sites.dat file. Use the user ID and PIN as your username and password.
Thanks gentlemen, all good. I needed this: "Put the client_id in the sites.dat file. Use the user ID and PIN as your username and password."
Deletejmf
I added another note to the USAA settings ;)
Deletehttps://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
-ameridan
ameridan, that's a great resource for people that are looking for it. I would suggest adding the link to https://www.usaa.com/accessid and how to use it. As far as I know, the link isn't published anywhere (it was an internal link from within the Quicken software).
DeleteDone.
DeleteIn response to Robert I got a unique client_id in the USAA url.
DeleteHowever I'm having no luck with getting into USAA using pocketsense (Jan2020) and the USAA provided "quicken" login info
Looks like I need to DEV version for USAA...
DeleteThis is my Target site entry for example:
ReplyDeleteSiteName : TARGET RED CARD
AcctType : CCSTMT
fiorg : Target
fid : 3820
url : DOWNLOAD ONLY
bankid :
brokerid :
appid : QWIN
appver : 2300
ofxVer : 102
mininterval:
timeOffset :
scrubber : scrubTARGET
PocketSense loads scrubTARGET.py at runtime only when a TARGET ofx file is to be scrubbed.
It loads scrubTARGET.py:
import os, sys, re
import site_cfg
from datetime import datetime, timedelta
from control2 import *
from rlib1 import *
from scrubber import scrubPrint
stat = False #global used between re lambda subs to track status
def _scrubber(ofx, accType='CCSTMT'):
global stat
ofx_final = '' #new ofx message
stat = False
if Debug: print('Function _scrubTARGET(OFX) called')
# dev: insert a line break after each transaction for readability.
# also helps block multi-transaction matching in below regexes via ^\s option
p = re.compile(r'()',re.IGNORECASE)
ofx = p.sub(r'\n', ofx)
# Target includes mutliple illegal ... <\hdr> headers in its ofx
pattern = r'[^<]+|[^<]+'
p = re.compile(pattern,re.IGNORECASE)
ofx_final = p.sub(lambda r: _scrubTARGET_r1(r), ofx)
if stat:
scrubPrint("Scrubber: removed bad tags in TARGET")
if Debug: print(ofx_final)
return ofx_final
def _scrubTARGET_r1(r):
global stat
stat = True
if Debug: print('Removing :' + r.group(0))
return ''
and passes the scrubbed ofx back to the rest of the existing scrubbing routines. If no custom scrubber is defined for a site entry, nothing is loaded or executed.
There is no need for multiple custom scrubbers as you can simply chain them UNIX style.
This is the code needed in scrubber.py:
# Load user-defined scrubber module
# defined per site via 'scrubber:module' in sites.dat
# stored as list of [ moduleName, loadedModule ]
# [0] : moduleName
# [1] : loaded module
# user-define scrubber must contain a function named "_scrubber" that will
# be invoked as _scrubber(ofx, accType)
if scrubberFile != '':
# check if already loaded
for thisScrubber in loadedScrubberList:
if thisScrubber[0] == scrubberFile:
# invoke scrubber with arguments (ofx, accType)
ofx = thisScrubber[1]._scrubber(ofx, accType)
break
# reached end of list without finding, so add it
else:
try:
my_module = importlib.import_module(scrubberFile)
# make sure it has _scrubber()
if "_scrubber" in dir(my_module):
scrubPrint('Scrubber: Loading scrubber from "' + scrubberFile + '"')
thisScrubber = [scrubberFile, my_module]
loadedScrubberList.append(thisScrubber)
# invoke scrubber with arguments (ofx, accType)
ofx = my_module._scrubber(ofx, accType)
else:
scrubPrint('User-defined scrubber "' + scrubberFile + '" missing _scrubber()')
except:
scrubPrint('Failed to import user-defined scrubber "' + scrubberFile + '"')
There are about seven added lines in load_sites in site_cfg.py to support the sites.dat scrubber line, and to load scrubbers for DISCOVER and TROWEPRICE by default since they were originally hard-coded in scrubber.py.
The changes are minimal and can be added to the latest version.
I have many other non-trivial changes that are not as straight-forward to add,
Missed this line:
Deleteimport importlib
scrubberFile = FieldVal(site, 'SCRUBBER')
Anyone have a decent bank that allows for direct download of multiple accounts?
ReplyDeleteI went from ING Direct to Capital One 360 to Discover Bank as each one shut down their direct download OFX import. Have still not found a replacement for Discover Bank. Would LOVE to not have to manually make entries anymore. Also do not want a bank I have to log into and then download from there and import, want it to be all direct with the same script.
I current direct import
American Express
Chase Visa
Chase Amazon Card
Fidelity
Etrade
Vanguard
and stock quotes
all work perfect and happy to give my info out connection info if anyone needs it.
Wishlist of other less used accounts (other than a new regular bank which is more important than all these combined in terms of saving time)
Target card
Discover Card (used to work have since stopped, and I pretty much stopped using because of it)
Janus Henderson
The American Funds
Chase works great. PNC too.
ReplyDeleteAnonymous March 10, 2021 said
ReplyDelete"Anyone have a decent bank that allows for direct download of multiple accounts?"
and later said
"Wishlist ... Janus Henderson"
Janus works fine for me, but I have not made any changes since long before they combined with Hendrerson. Here is the descriptor I use:
<site>
SiteName : Janus_Inv
AcctType : INVSTMT
fiorg : Janus
fid : 6115
url : https://ofx3.financialtrans.com/tf/OFXServer?tx=OFXController&cz=702110804131918&cl=50900132018
bankid :
brokerid : dstsystems.com
appid : Money
appver :
mininterval:
timeOffset : # my Janus uses scrubber, but would probably use 0 even without.
</site>
One important item: the account number you put into PocketSense is the fund number, comma, the 9-digit account number. No spaces. The fund number is two digits for me.
Also on https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
I do run a Janus scubber, which I would be glad to provide. It could affect the closing prices recorded by Money. I have not paid attention to it for a long while.
You might want to use a name for people to use when replying.
Cal Learner
Hi Cal,
DeleteOk thanks for the info. So when I enter in the info it connects and gives me a list of account numbers in the format you talked about. That all works fine and I had to enter in the info in setup 6 times (once for each account). However then when it comes over to import into money since the account numbers are different on each download (ex: 11,1234567 22,1234567 ) it will not keep matching to the one Janus brokerage fund I have money because each time I link it with each download it changes the account number.
So somehow that fund prefix needs to be removed from the download I suspect you are doing that in the scrubber.
I manually did that and reimported and each one correctly does the one fund but then sets the next one at qty zero? I have not looked into it to much but maybe this is fixed on your scrubber.
If you could provide that and how you call it that would be a huge help!
Thanks
Jason
DEV version updated.
ReplyDelete1. The following sites.dat options have been added:
promptStart: Yes #prompt/pause to continue when starting getData
promptEnd : No #prompt/pause to continue when getData is finished
2. Support for custom scrub modules has been added. It's a simpler implementation than Andrew mentioned, but consistent with other scrub routines. The Discover and TRowePrice routines have been moved to separate files, and are included in the base distribution (zip) file. These two provide ~ complete examples, and there's also a scrub_example.template file with a very simple description. Any scrub_*.py file found in the same folder as the base files (setup.py, getdata.py, etc) will be loaded and run for each statement, so it alone decides whether to process a statement or pass it back unchanged. The scrub module must contain a function named scrub([string]ofx, [string]siteURL, [string]acctType), and return a full ofx message.
- Robert
Haven't tried the latest DEV yet, but on the previous, I am noting that MS Money is having difficulty matching already downloaded transactions. These are transactions that were previously downloaded by the DEV version, get re-downloaded and flagged as new transactions. I then re-match them to the appropriate transaction.
DeleteThis would happen but only rarely with the last release version, before USAA changed. Now it seems to be pretty frequent.
Any suggestions?
The mismatching is only with the USAA transactions.
DeleteI haven't experienced any mismatching with USAA transactions (checking/savings). Do you have a scrub routine that might be changing the financial transaction ID? Some troubleshooting -- try to import the exact same OFX file and see if Money matches correctly, if not, then you have a problem with MS Money (or the OFX file could have flaws), you could also start a new, blank Money file and try to import your transactions there for a bit to see if you have something wrong with your Money file (could also try to run a repair on your Money file).
DeleteRon:
Delete1. When you see duplicates, I suggest that you accept and void the duplicates. That way you don't have to put in new info for the transactions you have dealt with, but Money will remember to ignore that FITID for that account in the future.
2. Start keeping copies of the USAA*.ofx files from the xfr folder. An alternative way to keep one is to rename the file to USAA*.xxx or somthing. Pocketsense deletes existing *.ofx before the regular (non-test) download of new. The point would be to let you compare the FITIDs of the same transaction in different downloads. This would enable the troubleshooting that Andrew Shafer suggests, and if the problem does not happen again, you could ignore or delete the *.xxx files.
3. The other thing that could happen would be for the OFX to start X-ing out some of the account number characters. I had that problem with a web download for a while. There could be scrubber relief.
4. If there is a recurrent changing FITID problem, there is a potential scrublet workaround-- always synthisize the FITID from payee, date, amount, and looking for transactions in the same download where each of those things match. There would be a time of transition, but after that you would not be bothered by their changing FITID bug. The OFX specifications require that the FITID be the same each time the same transaction is downloaded.
Cal Learner
Thank you all. I have run a level 2 repair on my mny file (no issues were found), and I have saved the last USAACC ofx file in a separate location. I'll keep saving until I get another mismatch, and then compare FITIDs. The most annoying issue comes when I overlook a duplicated transaction, and then wind up with a reconciliation problem. I will see what happens.
DeleteCal,
ReplyDeleteIt seems to be a problem with the FITID. Thanks for pointing me in that direction.
I can't be sure yet, but it seems that when a (some?) transaction is first downloaded, it has a different FITID than on subsequent downloads.
For example, two transactions that I first downloaded yesterday have FITID:
MDMvMTAvMjEoJDEzLjc2KTI0NDI3MzMy
MDMvMTAvMjEoJDQ5LjYzKTI0NDQ1MDAy
but the same transactions downloaded today have FITID
519FB6848F8569B287BF3F881DE81C02
EF7A67DAB59EEF693D2B34ECDA3FCC5D
None of the older transactions, which were seen as old transactions, had FITID's that started with "MDM"
If I understand what's going on, I don't think that deleting the duplicate will work since that one seems to have the stable FITID. I'd have to delete the original, wouldn't I?
Better would be if USAA corrected this OFX bug. I wonder how Quicken deals with it?
Hmmm. Looking at Quicken thread at "https://community.quicken.com/discussion/7887428/usaa-transaction-downloads-pending-transactions-edited/p4", it seems they have been having similar FITID related problems. There is a recent post dtd 11-March that reads: "I got a callback today from customer service acknowledging the issue as a USAA Critical Priority and Tech Support is working it." So hopefully it will be fixed for me, too, in the near future.
ReplyDeleteRon said "If I understand what's going on, I don't think that deleting the duplicate will work since that one seems to have the stable FITID. I'd have to delete the original, wouldn't I?"
ReplyDeleteVoiding one or the other would cause Money to ignore either FITID in the future. Since you may have already modified the older one with Category or Memo, it may be easiest to Void the newer.
Changing the interval may reduce overlap until things are settled. I see there is promptInterval choice in PocketSense. I have never used it. This is described in
http://pocketsense.blogspot.com/2013/01/jan-2013-version-of-scripts-available.html
Andrew Shafer described the "financial transaction ID" as likely the problem source. That is what FITID stands for.
Cal Learner
Thanks Cal, and Andy.
DeleteI think to avoid the problem, I'd have to not download very recent transactions. But is seems to me that a possible interim solution would revolve around ignoring transactions with a "strange" FITID -- eg those starting with "MDM" or maybe just FITID's that are not Base16 values. I may have to start learning a bit of Python and how to write a scrubber routine.
Ron said "possible interim solution would revolve around ignoring transactions with a "strange" FITID -- eg those starting with "MDM" or maybe just FITID's that are not Base16 values."
ReplyDeleteThe problem is that you might miss transactions that way. Will there always be a previous transaction with a non-MDM FITID?
If you wanted something that did that, I could write it, but I suspect that the downside might be more than the upside.
Cal Learner
Cal,
ReplyDeleteIt seems to be the case that the initial download of a recent transaction is assigned an "MDM..." FITID, and that subsequent downloads of that same transaction are assigned a HEX number FITID. For example, in my most recent download, only the two most recent transactions have "MDM" FITID's. At the next download, I expect these transactions will have changed to "normal" FITID's. (and the same was true for the previous day-to-day comparison). So I expect that by ignoring all "MDM" FITID's, I may delay the download of a few transactions, but I don't think I will miss any. (Of course, if USAA is "working on this", who knows what scheme they will come up with during the course of trying to repair this issue). I don't see a downside to ignoring the MDM transactions with the way they are doing things now. But I'd be interested in such a scrubber if it isn't too much trouble. With the latest DEV, I'd just have to add the module as a separate file in the folder. Of course, USAA may fix it soon making things moot. Thank you.
Is it possible that the MDM transactions are actually "pending" transactions? Those shouldn't be in the ofx statement, but maybe are?
DeleteRobert,
DeleteThat is my thought, also. And, apparently, some on the Quicken forum have documented that, but others say it has stopped. If I inspect my USAA page minutes after doing the download, there are no pending transactions. I should really check before I do a download to check that, though.
Robert,
DeleteThis morning I had five pending transactions and none of them were downloaded.
The two MDM transactions from yesterday were downloaded with different FITID's and showed up unmatched in MSMONEY.
I'd guess USAA will fix it, but I added a small scrublet, which is actually just a minor rewrite of the "remove zero trans" routine. I haven't tested, so it may need tweaks. I put it here.
DeleteAnyone that wants to share custom scrub routines... it's better to place somewhere w/ a link rather than posting code here, since blogger removes chars/indents/etc. I can add anything provided to the location above, but likely can't test.
Robert,
DeleteThank for that scrublet.
Examining it, it appears it will be testing TRNUID for MDM, but I think it should be testing FITID.
I will fiddle.
Thanks.
Ron
Ooops, got ahead of myself. I (very) quickly threw it together and had TRNUID in my mind. Changed TRNUID to FITID, with one minor change I noticed when editing.
DeleteRobert,
DeleteOK now it makes more sense to me. I'll give it a try.
By the way, if I wanted to restrict it just to my credit card account (the FITID's for bank accounts are very different) do I change: "if 'usaa/access.ofx' in siteURL:" to "if 'usaa/access.ofx' in siteURL and `CCSTMT` in accType:" ?
Robert,
DeleteI added the "scrub_usaa_fitid.py" file (unmodified from your last version) to the folder where all my other pocketsense files are located. But it did not filter out the "MDM" transactions. I tried using both `GETDATA` and also the `TEST` an account routines. There were still four transactions whose FITID's started with MDM in both the USAA and the combined.ofx files.
I confirmed that the regex returned the FITID in capturing group 2 (1-based numbering).
Not sure how to troubleshoot further.
Made a slight change... try again.
DeleteThanks. I will test it in a few days. I tried this afternoon, but the 'MDM' transactions are now being downloaded with "normal" FITID's, (and there are no new transactions with 'MDM' FITID's) so not a good test.
DeleteRon said "scrub_usaa_fitid.py"
DeleteIf the sites.dat entry for USAA says "SiteName : USAA" then I think you have to rename that scrub file to "scrub_usaa.py"
I am planning to post a scrub routine that accomplishes the same thing in what is intenede to be a more user-modifiable way.
Cal Learner
As currently written, there is no association between filename and function. Any file like 'scrub_*.py' in the root folder is loaded (s) and a single call is made to s.scrub(). The module must contain a function named scrub([string]ofx, [string]siteURL, [string]accType) and must return [string]ofx.
DeleteRobert: are you saying that the USAA scrubber could be named scrub_xyx.py? And that if there are five 'scrub_*.py' files, they each get called for for each account downloaded?
DeleteIf that is the case, it is up to the scrubber to test to see if it should change ofx, and otherwise return ofx unchanged.
Cal Learner
That's correct Cal.
DeleteI just did the pip on my python.
DeleteI have confirmed that all 'scrub_*.py' files get processed for each download. That will give maximum flexibilty, but will requiree that the scrubbers not get greedy.
With the Dev code, my Citicards stopped working, but still works with the older code. I got <MESSAGE;gt;We've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement. As a reminder, only credit card accounts can be linked.
Returned file says "VERSION:102". I will be trying to figure that out -- why it works with the old but not the new.
A test with a Fidelity account worked.
Cal Learner.
Cal: Are you running the dev scripts using the same ofx_config.cfg and connect.key files? I ask, because it sounds like Citi previously authorized access, but now sees a different clientUID tagged w/ the user/pw combo.
DeleteMystery solved. For dev, I made a new "ps_dev" folder. I had copied sites.dat and ofx_config.cfg to the new folder. Once I copied the connect.key file also, the the problem with Citicard access went away. So while I had never gone through that "relink" thing, but obviously Citicard accepted the first connectKey it saw, and remembered it.
DeleteOne thing I did in my ps_dev folder is to create psd.bat. It only does setup.py, but now psd shows up in the command prompt window header. That can reduce my confusion when comparing stuff.
One difference I see in running setup.py is that the new code only shows the ConnectKey on some accounts when listing accounts (after answering Y to "Show account connection keys?".
Cal Learner
Only accounts using ofxVer=103 will have a connect key (clientUID).
DeleteTrivial playing around file: scrub_xyz.py
Deletehttps://filebin.net/oqw6649s9z8h8gv0/scrub_xyz.py?t=phsq9m37
439 bytes.
Cal Learner
Robert said "Only accounts using ofxVer=103 will have a connect key (clientUID)."
DeleteThanks! My citicard sites.dat does contain "ofxVer : 103".
The returned OFX header says VERSION:102 which may have confused me. Clearly CitiCard cares about the connect key.
My Trivial playing around file: scrub_xyz.py file download URL evaporates in a week. One thing I found, (wasn't thinking)as I used the scrub_example.template is that I needed to put parantheses for scrubPrint().
Cal Learner
I have a USAA MDM removal scrublet available, but it uses a function that may be useful for many.
DeleteFunction remove_unwanted_OFX_transactions1(ofx,transactionType,findString)
That is the hightlight, and for the USAA scrub, it removes all
ofx=remove_unwanted_OFX_transactions1(ofx,"STMTTRN","MDM")
#remove transactions with FITID starting with MDM.
I suggest that the transactionType not have wildcards. I have not tested that, and I might modify this for safety to remove asterisks, which are often involved in greedy behaviour.
The findString is pretty unrestricted. It only acts with each transaction and determines if transaction is kept or discarded.
My earlier remove_unwanted_OFX_transactions() function could do a whole list of removals. But it was simpler to have multiple removals to require multiple calls with different parameter.
The split method was chosen because it was more foolproof. My tedious obscure regex strings designed to avoid greed would get greedy. Code could use some more sanity checking in case the OFX supplied is unusual.
Please comment on shortcomings or desired improvements. As it sits, it should do nothing except for USAA. If you un-comment line 59, it will operate on all OFX files. That is how I tested.
As an aside, I find that my 0x0d0a end of line sequences turn into 0x0d0d0a in the final *.ofx. Looks funny in some editors such as Notepad++, but Money accepts it.
I put this into a zip file named CL0.zip. https://filebin.net/c2lrzxcruycoid0j/CL0.zip?t=i3xgr0j5
This link will expire in a week. Grab it if you like.
Cal Learner.
Cal, I downloaded, as someday I would like to be able to only download transactions from investment accounts, and ignore positions with daily prices. As far as I'm concerned, CDs should remain at par (100.00) or at the price you paid, and not change daily. Example:
Delete...
Robert,
DeleteYour USAA scrubber works. I tested today and there are MDM FITID's that get removed if I run the scrubber. Thank you.
Can I limit it to just the USAA CC account by changing
"if 'usaa/access.ofx' in siteURL:" to
"if 'usaa/access.ofx' in siteURL and `CCSTMT` == accType:" ??
or is that improper syntax?
Yes, although I'd write it a bit different:
Deleteif 'usaa/access.ofx' in siteURL.lower() and accType=='CCSTMT':
I'd guess that no other account type will have FITID start with 'MDM', though, so it may not matter.
Thank you. And that 'lower()' is what you have in your modified scrubber when the original didn't work. I'll have to remember that. You are probably correct about the other USAA accounts not having MDM issues.
DeleteHas anyone used this with TIAA? It is able to login and get the account list but when trying to download transactions it gives a 2003 account error. I compared the data being sent with that from MoneyDance which seems to work and it is pretty much the same. Only difference seems to be the trnuid is formatted differently.
ReplyDeleteHi Cal I figured I would repost this in case you missed it up above in the replys on another thread... would love to get this working with Janus.
ReplyDelete----
Ok thanks for the info. So when I enter in the info it connects and gives me a list of account numbers in the format you talked about. That all works fine and I had to enter in the info in setup 6 times (once for each account). However then when it comes over to import into money since the account numbers are different on each download (ex: 11,1234567 22,1234567 ) it will not keep matching to the one Janus brokerage fund I have money because each time I link it with each download it changes the account number.
So somehow that fund prefix needs to be removed from the download I suspect you are doing that in the scrubber.
I manually did that and reimported and each one correctly does the one fund but then sets the next one at qty zero? I have not looked into it to much but maybe this is fixed on your scrubber.
If you could provide that and how you call it that would be a huge help!
Thanks
Jason
Jason: I just checked my Janus downloads. The ACCTID are of the form
ReplyDelete>ACCTID<41,123456789
I do run that through scrubber that puts that on its own line.
So I tried again with the dev code. I renamed scrub_*.py as xxscrub_*.py to disable those. Same result. In fact, the ACCTID was on its own line.
I don't know what to say.
In your XFR folder, try this command in a command window:
find "ACCTID" jan*.ofx
What do you see?
Cal Learner
---------- JANUS_INV220210322102754555961.OFX
Delete48,XXXX89691
---------- JANUS_INV220210322102754854334.OFX
50,XXXX89691
---------- JANUS_INV220210322102755630762.OFX
54,XXXX89691
---------- JANUS_INV220210322102755951002.OFX
62,XXXX89691
---------- JANUS_INV220210322102755997534.OFX
74,XXXX89691
---------- JANUS_INV220210322102756916068.OFX
76,XXXX89691
Note I added the XXXX.
This shows the problem I am dealing with. Since they are adding the fund number in front of the account number I cannot match each fund to the one janus account I have in money.
If I manually match the first one on import it sets the janus account number to 48,XXXX89691. Then it moves onto the second one which of course does not match to 48,XXXX89691 and asks again what account this OFX is for.
I need a way for it to remove the fund number prefix in each file so the account number is just XXXX89691 and I can set the money account number to XXXX89691 and then it should just match each file and import.
Its not a huge deal in terms of transactions on this account, I can keep doing it manually if I need to for the rare trade or adjustment. But I figured I would ask if someone else has already dealt with this problem.
Janus user with 6 accounts:
DeleteNice description. I understand what you want and need to accomplish this. I can make a Janus scrubber that will consolidate into one account. I will run with the PS DEV code.
I have two Janus accounts in Money. I can understand why you will want 6. I think I originally had one account, and just moved the transactions. However I got tired of that, and gave in to separate accounts.
My intention is to make a function that turns the initial two characters of the account ID to a fixed thing. My initial thought is to make them into actual X characters. My intention is to make this a bit general to handle the case where the FI itself turns the first several characters into Xs but switches back and forth. I have seen that in the past, tho the FI settled down and stopped doing it.
I will expect I will also add a date scrubber function that works nicely for me with Janus.
Cal Learner
Janus user:
Deletehttps://filebin.net/3ycqtdvlxs1pz82u/cl2.zip?t=6n3zuax1 has CL2.zip which contains scrub_janus.py Disappears in a week.
Defines a function named
Subsitute_first_ACCTID_chars_ofOFX_with(ofx,sub_string)
The second parameter is XX in when called by the scrubber, and those characters replace the first characters rplace the first characters of ACCTID. So could be used for more than Janus.
Cal Learner.
Still has the stuff I tried for USAA, but that will not interfere.
Got it. Works perfect for the account number problem.
DeleteI still have the problem of the fact that there is only one fund on each statement money wants to update the shares of another fund to zero since it does not see it on the statement. It is very strange as after I import each one it only selects one fund that it wants to say I have zero and it seems to be random which fund it picks.
Makes no sense. The statement only has the one fund in it and the data is perfect for that import. It would make more sense to me if money wanted to update all the other funds to qty 0 after importing one of these files since it only has record of one fund on each import. But no its always just one fund.
Just checked the new DEV scripts. I previously had been using the 04-Jan-2020 scripts with some custom scrubs. Here are my results:
ReplyDelete* My regular and retirement accounts with large, well known banks all worked fine.
* Target and Trans-America downloaded QFXs processed nicely with the new scrubber updates. No need for my custom scrub any more.
Only one major problem:
* E-Trade is returning a null statement. I reran with the old scripts to make sure it wasn't a server issue. There is something in the new scripts that is breaking E-Trade. I copied my sites.dat, connect.key and ofx_config files directly from the old folder to the new folder. I tried getting account information from E-Trade using setup. Old scripts properly provide an account. New scripts say an error occurred.
@Carlos: If you have a chance, can you run each in debug to see what (exact) differences may be? In particular, the request header fields and the ofx request itself. - Robert
Delete@Robert:
DeleteThe actual OFX requests (including OFX Headers) are the same except for and so I think there is no issue there.
The old scripts have "\r\n" in the debug output whereas the new scripts have an actual carriage return therefore I can't do a one-to-one comparison on what is actually being transmitted, but I assume that the carriage returns are handled the same in both cases. I can dig more into this if we think that is where the issue lies.
I noticed that the new scripts have a repeated SENT:/RECEIVED: section. Do you send the request twice?
Here are the actual HTML sent headers:
OLD SCRIPT:
POST /cgi-ofx/etradeofx HTTP/1.1\r\nContent-Type: application/x-ofx\r\nHost: ofx.etrade.com\r\nContent-Length: 703\r\nConnection: Keep-Alive\r\n\r\n
NEW SCRIPT:
{'Host': 'ofx.etrade.com', 'Connection': 'Keep-Alive', 'Content-Type': 'application/x-ofx', 'Content-Length': '703', 'User-Agent': 'InetClntApp/3.0'}
* I notice the new scripts don't have the cgi-ofx portion of the URL, but I'm guessing that it is not being printed and is not the issue since other sites work just fine.
* main difference that pops out is the addition of User-Agent
Here are the actual HTML received headers:
OLD SCRIPT:
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 25 Mar 2021 13:24:55 GMT
header: Content-Type: application/x-ofx; charset=UTF-8
header: Transfer-Encoding: chunked
header: X-ION-HOP: 1
header: Server: Apache
header: Via: 1.1 google
NEW SCRIPT:
HEADER: {'Date': 'Thu, 25 Mar 2021 13:22:40 GMT', 'Content-Length': '0', 'Via': '1.1 google', 'Content-Type': 'text/plain; charset=UTF-8', 'X-ION-HOP': '1'}
Right away I noticed that the COntent-Length is 0 and the Content-Type is text/plain, which I would guess means the request wasn't properly processed.
The new default user agent is not liked by E-Trade. Setting up a site-specific user agent of "none" or "PocketSense" resolves the issue.
DeleteI have a request/recommendation for handling manually downloaded files. For most of us a manual download will always end up in whatever folder the browser normally downloads to. Therefore an extra manual step is needed to copy to the importdir. It would be nice if Pocketsense knew to look in the download directory (set in sites.dat) and grabbed any OFX or QFX files from there instead.
ReplyDeleteCarlos:
ReplyDelete1. Well done on figuring out that E-trade is picky about the user agent, and setting up a site-specific user agent of "none" or "PocketSense" resolves the issue.
2. You can invoke GetData.py from a batch file. That bat file could include moving any *.ofx, *.qfx etc files to where PS looks for them.
My batch file does several things, including CD to the right folder, logging some info, and more. My batch file gets invoked by a Windows scheduler, but you could certainly make a file to invoke manually.
Cal Learner
I have a biggish scrub_cl3.py file in scrub_cl3.zip posted at https://filebin.net/l4nc5y4dm7d18zml/scrub_cl3.zip?t=0qt7st2n
ReplyDeleteThis link will stay active for a week, but I may make an updated version later.
Not only does this have working and tested scrubbing that works for the PS dev software, it defines some functions that people might want to use in their own scrublet. Here are some of the functions that may be of general interest:
def _scrubDateFix(ofx):
#Copy good timestamp from first DTPRICEASOF to DTASOF date if needed,
#Avoids wrong price history in Money such as Sat, Sun, holiday dates.
#if change is made, scrubPrint before and after value
#if no DTPRICEASOF element in module, does nothing but a scrubPrint
#This is for investment accounts, and not bank, CC, etc.
#This is the best of the bunch with very general application I think
#works for Janus and Vanguard and probably others
def _scrubDateFixToWeekday(ofx):
#if you download Fidelity on Sunday or Monday, you will probably get an incorrect date in the Money history. Try it. This function moves the date for transactions back to the preceding day, or to Friday if this is Sunday or Monday. I have been using this method for quite a while, and have moved it into the scrublet method. I changed the name to take Fidelity out of it, and to recognize that it can be used for others that don't put DTPRICEASOF values that reflect the closing time. With this methodology, I suggest that you not download the day after a holiday; not a big deal.
def scrubDateFix_skimpy(ofx):
#same purpose as _scrubDateFix but less robust, Do not use both.
#this is the same method I have been using for some FIs
def _Subsitute_first_ACCTID_chars_ofOFX_with(ofx,sub_string):
#called with second parameter.
# can use sub_string "XX" for Janus to combine accts, or "XXXXXXX" for some other.
#Made to be more general purpose.
#The current name should be something better. Make suggestions
def remove_unwanted_OFX_transactions1(ofx,transactionType,findString):
#one of my favorites. Removes transactions of a specified type that contain a specified string. I have been using a function that will do more from a tuple defined in the source, but decided this would be better for most people. For my Fidelity scrubber I call this 3 times with different parameters.
A prettier version of these could be candidates for a utility collection of functions that could be in module used like a function library IMO. I hope you find something useful there. Tested for Fidelity, Janus, Vanguard, and CitiCard
Cal Learner
Cal: My intention is to have a folder of "scrublets" that folks can use as they wish. I'll add whatever you post. It looks like this is an extension of the last one. Is that a fair assumption... meaning you're putting everything in one module? -Robert
DeleteSuper. I just uploaded scrub_cl4.zip which contains scrub_cl4.py
DeleteIt supercedes scrub_cl3.py with the main improvement in the _prettyOFX(ofx) function. Seems to be working well, and you could build the function into the scrubber.py if people think it is general enough.
Yes, right now it is all-in-one for the FIs I use. It should be easy enough to change into a single-FI scrubber. I think of it as a development thing that others might adapt parts of.
I renamed scrub_cl3.py to xxxscrub_cl3.py in my folder to make sure I don't run the older version and the new scrub_cl4.py both.
https://filebin.net/o1htqmm8b12iq128/scrub_cl4.zip?t=k97vrqo4
Cal Learner
I posted a bug fix version of scrub_cl4.py development code. Download link is
Deletehttps://filebin.net/ykcsb24ujzrj48sp/scrub_cl4_20210331.zip?t=9mwqrjkw
scrub_cl4_20210331.zip contains two files: scrub_cl4_20213032_notes.txt and scrub_cl4.py and replaces previous files. Much of the code is trying to apply the proper dates to the Money price history. I think this works well always for Janus, and for Fidelity and Vanguard if some time windows are observed.
For Fidelity avoid between midnight et and 5am et.
For Vanguard avoid between 4pm et and an undetermined time to at least midnight et.
So morning and early afternoon is best. I run a Windows scheduled task at about 6am et to automate it.
No known problem with Janus downloads.
If you don't really care if your date history is off by a day, don't worry about the preceding.
Cal Learner
This comment has been removed by the author.
ReplyDeleteI got USAA working again with this version. Thanks so much!
ReplyDeleteI saw that USAA was outputting check number "0" as a default on every transaction, which was annoying. Following some other scrubbing examples, I came up with the following module, which I called "scrub_usaa_check_numbers.py". I'm pasting it here in case anyone finds it useful (or wants to improve on it—I'm not a Python programmer).
# Remove check numbers with a value of "0" from USAA transactions
from control2 import *
from scrubber import scrubPrint
import re
def scrub(ofx, siteURL='', accType=''):
if 'USAA' in siteURL: ofx = _scrubRemoveCheckNumberZero(ofx)
return ofx
def _scrubRemoveCheckNumberZero(ofx):
ofx = re.sub('<CHECKNUM>0<', '<', ofx)
return ofx
Robert,
ReplyDeleteOut of curiosity, are there any plans to update the code to Python 3 now that Python 2 is "officially" (whatever that means) dead?
Carlos
Carlos: Development of Python 2.x is stopped, including bug fixes. We could also say that Python 2 is no longer a moving target; that has some advantages. There are various 2.7x versions still available for download. Python 3 is still moving. :-) So porting may be a mixed blessing.
ReplyDeletehttps://www.zdnet.com/article/programming-language-python-2-7-code-is-now-frozen-last-release-coming-in-april/ says "While Python 2's end of life has been a long-time coming, it's expected that some industries, such as finance, will continue to use Python 2.x code for years to come."
There is another thing to consider: OFX 2.x may be adopted by some FIs. I have incorporated your _scrubRemoveCheckNumberZero(ofx) code into my potential next version of scrub_cl4.py. However I modified the code to use this line:
ofx=re.sub(r'<CHECKNUM>0[\s]*(</CHECKNUM>)*[\s]*<', '<', ofx) #works even if closing tag or whitespace added
That will also work with OFX 2.x and will deal with potential newlines.
Cal Learner
After a period of no problems, I again noted duplicate transactions in this mornings USAA Credit Card download. Inspecting yesterday and today's OFX files, it seems they have changed the "bad" FITID from 'MDM*' to 'MDq*'. I changed the scrublet to look for just 'MD', but I sure wish they would get their act together.
ReplyDeleteHelp please! So I'm here trying to fix USAA downloads. I'm running Python 2.7.15, I downloaded the DEV files from the link above, and copied the new files into my PocketSense folder. I must be doing something wrong, because when I run setup or getdata, they briefly flash and close a window. The windows say:
ReplyDeleteTraceback (most recent call last):
File "D:\Program Files (x86)\PocketSense for MS Money\Getdata.py", line 56, in import ofx, quotes, site_cfg, scrubber
File "D:\Program Files (x86)\PocketSense for MS Money\ofx.py", line 79, in import requests, collections
ImportError: No module named requests
Traceback (most recent call last):
File "D:\Program Files (x86)\PocketSense for MS Money\Setup.py", line 14, in # 28Aug2013*rlc
File "D:\Program Files (x86)\PocketSense for MS Money\ofx.py", line 79, in import requests, collections
PIP: See up top where it says "pip install requests".
ReplyDeleteCal Learner
Doh, "read the instructions", thanks! I installed 2.7.18 from ActiveState and installed requests...now getdata and setup are working. But now I don't have the IDLE feature to edit scrubber files. Do you know how I can reenable that?
ReplyDeleteCheck out Notepad++
Deletehttps://notepad-plus-plus.org/downloads/
To get the "Edit with IDLE" context menu back you might have to uninstall and reinstall (carefully look at the install options to get the context menu feature).
DeleteAlternatively, I recommend using Microsoft's VSCode for programming and development.
No luck getting IDLE after uninstall/reinstall, but notepad++ got me the editor I needed...Thanks!
ReplyDeleteAndrew Shafer, thanks for the Chase Amazon scrublet too!
I'm trying to learn this scrubber feature. Scrubber.py seems to be full of defs, but I created a separate Amazon scrublet (correct term?) Could you please tell me where does this go? Not sure what the 'scrub routine' is...
ReplyDelete===============
if 'CHASE' in siteURL: ofx= _scrubAmazon(ofx)
===============
LAG and all: In the PocketSense DEV version from February 24, 2021, any/all file placed into your Pocketsense folder that has the form scrub_*.py will be run by the scrubbing process.
ReplyDeleteI put such a file into a zip file named scrub_cl4_20210414.zip .
https://filebin.net/t42wzy3k5e1ngaw4/scrub_cl4_20210414.zip?t=8btfsm0v
is a link to that file, that will only stay around for 1 week. If you want a copy after that goes away, post that. I will post a new link, although that may or may not have changed a bit. At first I was making the py file have a different name each time, but by keeping the same name, it became less likely that it would exist at the same time as an earlier version. I am not sure that having two versions in the same folder would mess things up, it is not a good idea. The zip file contains scrub_cl4.py and scrub_cl4_20210414.txt. The *.txt file extends these remarks a bit.
scrub_cl4.py is currently intended to stand alone. It can, however just be used as an example of some code. If, for example, you were to rename it to xxscrub_c14.py, it would be ignored by PocketSense.
Cal Learner
FYI. If anybody is using Pocketsense to get transactions from Alright Systems (which processes 401K data for many large companies; used to be Hewitt Associates) and is having problems, you might want to try their new URL: https://seven.was.alight.com/eftxweb/access.ofx
ReplyDeleteThe fiorg value is still hewitt.com
Harold
Harold, does this look right? I added to my settings page:
ReplyDeleteSiteName : Alright Systems # (was Hewitt Associates)
AcctType : INVSTMT
fiorg : hewitt.com
url : https://seven.was.alight.com/eftxweb/access.ofx
fid : 242
bankid :
brokerid : Alright
appid :
appver :
mininterval:
I don't have anything specified for brokerid, but it may just ignore what is put in. Did it work for you?
DeleteHarold: NxtTek lists successful site descriptors on https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/ and now, with your info, there is an Alright Systems entry.
DeleteCal Learner
On a download today I get the following error for Fidelity:
ReplyDeleteInvalid OFX statement detected
** Review .\xfr\FIDELITY20210508105107596852.ofx for possible clues
Charles Schwab got the Invalid OFX statement error when they disabled the ability to download via the scripts. Is Fidelity doing the same thing or is this a temporary error that will get resolved?
Fidelity had been working up to last week (old version from 2019). Now gets "You don't have permission to access "http://ofx.fidelity.com/ftgw/OFX/clients/download" on this server."
ReplyDeleteI am also getting that error. I tried with both the release version of PocketSense and the Dev version.
ReplyDeleteThey both had a Reference# in the short OFX file.
My access Friday morning worked fine.
Cal Learner
I see that the History on the Fidelity account website has a Download link. That brings in a CSV file with 17 columns, some of which are empty:
ReplyDeleteRun Date,Account,Action,Symbol,Security Description,Security Type,Exchange Quantity,Exchange Currency,Quantity,Currency,Price,Exchange Rate,Commission,Fees,Accrued Interest,Amount,Settlement Date
I would like to think that Fidelity Investments would cooperate like USAA did. I am pessimistic of that.
Cal Learner
The MoneyDance entry for Fidelity shows what is really happening.
ReplyDelete{
"id" = "md:1117"
"access_type" = "OFX"
"fi_name" = "Fidelity Investments"
"bootstrap_url" = "https://ofx.fidelity.com/ftgw/OFX/clients/download"
"fi_id" = "7776"
"use_short_dates" = "y"
"fi_org" = "fidelity.com"
"broker_id" = "fidelity.com"
"app_id" = "QWIN"
"app_ver" = "2700"
"app_id" = "MDNC"
"app_ver" = "2021"
"use_ofx_certs" = "y"
"proxy_tunnel_host" = "ofxp.infinitekind.com"
"proxy_tunnel_port" = "80"
}
This is unlikely to be fixed without both code changes and cooperation from Fidelity. I doubt InfiniteKind would be foolish enough to have an open proxy.
FWIW, my Fidelity connection went through this morning.
DeleteSame here on both regular Fidelity and Netbenefits
DeleteSo for now, it might be best to adapt the Schwab adaptation of hleofxquotes (ver. 287 @ https://bitbucket.org/hleofxquotesteam/hleofxquotes/wiki/Build_20201206_287 ), and use that for Fidelity
ReplyDelete-ameridan
Ameridan, how would we use the hleofxquotes? If I'm reading @Andrew's MoneyDance settings correctly, MoneyDance is forcing the connection to Fidelity to go through their own proxy. Presumably ofx.fidelity.com will now only accept connections from "known" proxies. Without having access to one of these known servers, I'm not sure what hleofxquotes would do for me.
DeleteThere is a section for proxy settings:
ReplyDelete# proxy setting
http.proxyHost=
http.proxyPort=
hopefully the author will pipe in, as I don't have fidelity accounts, but try it out.
Fidelity Investment download is working!!!
ReplyDeleteI just gave it a try, and got proper operation.
I am pleased.
Cal Learner.
That is a relief! Hope it stays that way.
DeleteThis morning, for me, Fidelity is working, but USAA is not and the response includes "request unsuccessful". Hopefully it is due to USAA doing maintenance, and not some other issue.
ReplyDeleteUSAA worked for me just now, for 2 checking and 2 savings accounts.
DeleteIt does seem to be working again. I guess it was just a transient glitch.
Delete1. Things have been working great with PocketSense for me. I expect it is so for others, given how quiet things have been.
ReplyDelete2. I have a scrubber named scrub_cl4.py that works with the Dev version of PocketSense. My current file is dated as 04/11/2021. If somebody wants a copy, I could post it in an accessible location for a week. It covers the following FIs, but will do nothing if the OFX being processed is not one of these FIs:
a. Fidelity Investments (more sophisticated/complex than you would expect)
b. Janus
c. Vanguard (surprisingly there is a date fix for quotes dates)
d. USAA ( bank... I have not tested myself)
e. Citicard (not bank)
3. It would be easy to break that up into 5 different scrubbers. While they do share some local functions, those would not take a lot of space to replicate. The combined file is 11,508 bytes. The other way would be to modify the def scrub() to not test for FIs that are covered by a different scrubber. Some people don't feel comfortable modifying Python code, even if it is just deleting or commenting-out a line. The broken-out scrubber files might have a comment that shows the descriptor that I use in my sites.dat file.
4. I do use a Windows scheduled task to run PocketSense. Something that I have found is that browsers will occasionally disable my task that runs the batch file. I also am running another Python program that makes a sound if that task ever gets disabled. I only mention that for people who might be trying a Windows scheduled task and find it turning off on them unexpectedly.
Cal Learner.
First of all, I appreciate you develop the script to get quote from Yahoo to feed MS Money. I used the script a few years. I am new on Python programming. In fact, starting a class on python essential. Can you help/teach me how to convert this python code to 3.7 or higher compatible?
ReplyDeleteHeads up on Vanguard accounts.
ReplyDeleteIn the past 4 months or so I noticed two of my Vanguard accounts would not always show all transactions. Manual parsing of the OFX files showed the transactions were present, and seemed properly formatted. Further analysis showed that the number of shares, price per share, and purchase price were not consistent. This inconsistency was enough to cause MS Money to throw out the transaction as invalid (no error given)
I think it has to do with the way Vanguard computes the number of shares to purchase and then rounds it to the nearest one thousandth share, while retaining the share price at the time of purchase. Interestingly, if you look at the cost basis report in Vanguard, it has a modified share price that is consistent with the number of shares and total purchase price.
I'm going to try to write a scrubber to fix this issue.
Thanks for reporting Carlos. I too use Vanguard, but haven't (yet) witnessed the issue you're seeing. Just FYI for now...
DeleteCarlos: I will be interestest in seeing your scrubber. My Vanguard scrubber only tweeks the date used for the closing prices sent to Money.
ReplyDeleteI could see having two levels: warn and maybe log, and optionally repair.
It may be this could be used in other scrubbers as a sanity check that is not expected to be needed, but could warn of such an anomaly.
Cal Learner
This is a very interesting situation. I don't know why I haven't seen this error before, and now I'm suddenly seeing it a bit. But here is what I have discovered so far:
Delete* Vanguard always reports shared price to the nearest cent and number of shares to the nearest 0.001
* When the share price*shares = transaction total (when rounded to the nearest cent) money does nothing
* When share price (2 decimals) * shares (3 decimals) does not equal total (2 decimals) money will sometimes replace share price with a 6 decimal equivalent, while other times it rejects the transaction
I can't figure out why Money sometimes "corrects" the share price and sometimes rejects the transaction or why rejections only started happening in the past 3 months (I suppose it is possible prior to that Vanguard was reporting share price to more decimal places, but I doubt it)
>> Problem after upgrading from to 04-Jan-2020 from 24-Aug-2018.
ReplyDelete>> I changed the Download interval to 45 days from 30 in sites.dat and updated the comments but have not tried to diagnose the problem.
>> Here is what I am getting with account information replaced with Z's and X's:
PocketSense OFX Download Python Scripts, Ver: 04-Jan-2020
Download transactions? (Y/N/I=Interactive) [Y]
Password:
Download interval= 45 days
AMERICAN EXPRESS : ZZZZZZZZZZZZZZZ|XXXXX : Getting records since: 20210524
** An ERROR occurred retrieving POST response from online.americanexpress.com
Exception type :
Exception val : [Errno 2] No such file or directory: '.\\xfr\\AMERICANEXPRESS20210708122116566148.ofx'
HTTPS ResponseCode : 200
HTTPS ResponseReason: OK
COSTCO CITIBANK VISA : XXXXXXXXXXXXXXX : Getting records since: 20210524
** An ERROR occurred retrieving POST response from mobilesoa.citi.com
Exception type :
Exception val : [Errno 2] No such file or directory: '.\\xfr\\COSTCOCITIBANKVISA20210708122117220243.ofx'
HTTPS ResponseCode : 200
HTTPS ResponseReason: OK
Searching .\import\ for statements to import
No files were downloaded. Verify network connection and try again later.
Press to continue...
Grab the latest DEV version (link in intro to this post above).
DeleteI installed the latest DEV version and now Getdata and Setup terminate immediately.
ReplyDeleteIt looks like I am using Python 2.7.1 from ActiveState that I installed in 2015.
Definitely update to the latest V2.7: https://www.activestate.com/products/python/downloads/
ReplyDeleteAfter updating I am running ActiveState Python 2.7.18 Build 0000 64-bit Version effective date: March 25, 2019.
ReplyDeleteGetdata and Setup no longer terminate immediately, but Getdata still does not work:
AMERICAN EXPRESS : ZZZZZZZZZZZZZZZ|XXXXX : Getting records since: 20210525
** An ERROR occurred sending POST request to https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload
Exception type :
Exception val : [Errno 2] No such file or directory: '.\\xfr\\AMERICANEXPRESS20210709091205794556.ofx'
HTTPS ResponseCode : 200
HTTPS ResponseReason: OK
COSTCO CITIBANK VISA : XXXXXXXXXXXXXXXX : Getting records since: 20210525
** An ERROR occurred sending POST request to https://mobilesoa.citi.com/CitiOFXInterface
Exception type :
Exception val : [Errno 2] No such file or directory: '.\\xfr\\COSTCOCITIBANKVISA20210709091207259732.ofx'
HTTPS ResponseCode : 200
HTTPS ResponseReason: OK
I see this is omitting the Exception type, probably because it is enclosed in less than and greater than symbols.
ReplyDeleteException type : type 'exceptions.IOError'
Strange, but it appears the .\xfr subfolder is missing. Run Setup.py and test one of the accounts, then try getdata.py again.
DeleteThanks, that took care of the problem except that I received a Citi message shown below that I resolved by doing as the message instructed. I also verified that I was able to include OFX and QFX files in the .\import subfolder, which is why I was upgrading.
ReplyDelete"We've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement. As a reminder, only credit card accounts can be linked."
New problem -
ReplyDeleteGetdata terminates when I try to use it without any accounts defined to ofx_config.cfg and sites.dat.
I want to use the PocketSense ./import folder for OFX files where the accounts don't directly work with PocketSense. One of the accounts is a credit union checking account that stopped working with PocketSense several years ago, and the other account is a Capital One credit card that never worked with PocketSense.
Enable quotes in Setup. You can disable actual quotes being retrieved by removing (or commenting out) the STOCKS and FUNDS entries in sites.dat.
DeleteEnabling quotes did not help. I get the initial Download transactions request when I run Getdata, but Getdata immediately terminates when I reply Y.
ReplyDeleteThere is a Chase Bank checking account I could add if that is the easiest way to deal with this. I could also provide my Sites.dat file if you want to try reproducing the problem.
PocketSense OFX Download Python Scripts, Ver: 2021-Mar-12 DEV
Download transactions? (Y/N/I=Interactive) [Y]
Person with import problems: Make sure your files with the qfx files are in a folder named "import" in the same folder that your xfr folder and setup.py are in.
DeleteWhen ready to type setup.py, try this command:
dir import
That should show your qfx or whatever files.
If you want to post the contents of your sites.dat, note that you should replace any > character with these four characters: >
Replace any < character with <
Cal Learner
Cal -
ReplyDeleteI have successfully used the PocketSense import folder with a different Microsoft Money file where I have PocketSense set up to obtain transactions from an American Express credit card and a Citibank credit card. The Getdata script properly processes a Capital One QFX file that I place in the import folder.
I have not previously used Getdata with the Microsoft Money file where Getdata is failing, but I am using the same Sites.dat file except with the definitions removed for the American Express credit card and a Citibank credit card.
I will try uploading my Sites.dat contents with most of the comments removed.
Paul White
# SITES.DAT
Delete#General settings
#--------------------------------------------------------------------------------
defaultInterval: 45 #define default download interval (days)
promptInterval: No #prompt user for alternate download interval?
SaveTickersFirst: No #Send stock/fund quotes to Money first? Default = No
SaveQuoteHistory: No #Save quote history to QuoteHistory.csv? Default = No
quietScrub: No #Suppress scrubber messages (default=No)
skipZeroTransactions: No #Remove $0.00 transactions from downloaded statements
CombineOFX: Yes #Combine ofx files before sending to Money
#Warning: Do not enable CombineOFX until you test all account settings, and
# verify that data is loading correctly to Money accounts.
skipFailedLogon: Yes #If a connection to a site fails during Getdata, no further connections
#will be attempted for that site+username combo during the session.
#default = Yes
promptStart: Yes #prompt/pause to continue when starting getData
promptEnd : No #prompt/pause to continue when getData is finished
#SITE ENTRIES
#--------------------------------------------------------------------------------
<site>
</site>
#--------------------------------------------------------------------------------
# Stock quote options
#--------------------------------------------------------------------------------
EnableYahooFinance: Yes # Enable quote lookup on Yahoo Finance
YahooTimeZone: -5:EST # Timezone rule for the Yahoo quote server (default = -5:EST).
ShowQuoteHTM: No # Always show quotes.htm from Getdata
AskQuoteHTM: No # Ask to show quotes.htm from Getdata (overrides ShowQuoteHTM)
ForceQuotes: No # Force Money to record a transaction when importing quotes*
<stocks>
#--- indexes ---
#--- stocks ---
#--- Currencies ---
</stocks>
<funds>
</funds>
Paul: I am a little confused with your mention of multiple versions of sites.dat, including the stripped down one that you posted. I think you should keep all sites in your sites.dat for each use.
ReplyDeleteI suggest that you open a command window and CD to the folder that contains the import folder. Then run Getdata.py from that command window. That way there is no ambiguity of what the working folder is.
Rather than having more than one version of sites.dat, you might use different versions of ofx_config.cfg. Setup.py modifies that file, and it contains the list of things to do when you run Getdata.py. It also contains password info. Anyway, you could work out a scheme where you make a copy of ofx_config.cfg to say ofx_config.cfg.purpose1. Then with Setup.py, you would add or delete accounts. Exit Setup.py and make another copy of ofx_config.cfg to say ofx_config.cfg.purpose2. Then maybe have batch files that CD to the folder, and copy the particular ofx_config.cfg.purpose* over ofx_config.cfg, before running Getdata.py.
There are various strategies for invoking PocketSense. Mine involve batch files, but I had not considered fetching different things for different Money files. My daily batch file checks to make sure that the right Money file is open before proceeding, and if no Money file is open, to open the right Money file. Then wait for a while to allow that to happen.
My earlier mention of Setup.py was not useful. As far as I can tell, you cannot process the things in the import folder from Setup.py. I have not tried the import folder, because the ofx/qif files I download by web have not needed scrubbing... OK, I can think of one that could, but I work around that flaw by only downloading a specific date range of transactions into the OFX.
Cal Learner.
Cal - I use a batch script in which I select the specific Microsoft Money file I want to open. That batch script copies the associated ofx_config.cfg and sites.dat files into the PocketSense folder. I have been doing this for about 10 years and use separate sites.dat files because that allows me to have different parameters for different Microsoft Money files, which I find simpler than having all the accounts in the same sites.dat file.
DeleteI initiate Getdata with a shortcut from Microsoft Money > Favorites > Favorite Web Sites. The shortcut is named "Update Now…" to make it look like Microsoft Money worked before I started using PocketSense. There was a description of this technique in Ameridan's Blog.
I decided to try using the new PockeSense import function because I have some OFX accounts where I thought the scrubber might be helpful and also because I have 3 QFX-only accounts where it is more convenient than renaming the .QFX suffix to .OFX. I considered trying to modify the Windows Registry to treat QFX like OFX, but until recently I only needed to download the QFX files a couple of times per year, which seemed like less trouble than modifying the Windows Registry.
The PocketSense import function is working fine with the Microsoft Money file where I was previously using PocketSense, but I want to also use it with a Microsoft Money file where I need to manually download the OFX files but don't otherwise need PocketSense. I suspect I could make this work by defining an account to sites.dat and ofx_config.cfg, but I also suspect that the PockeSense script could be modified to work properly without any files.
Paul White
I'm not able to replicate the problem. I installed a fresh copy of the latest beta version, ran setup, and enabled quotes. I did not add any accounts. I then edited the new sites.dat file to remove the default quote tickers (stocks/funds). This step would be optional, but it's probably what you'd want. I then copied a recent ofx download to .\import and ran getdata. It ran correctly and imported to Money.
DeleteHello - it looks like python 2.7.18 from python.org breaks PocketSense. Last one to be tested and working out of box is 2.7.16 from python.org.
ReplyDeleteOn July 9, I updated to ActiveState Python 2.7.18 Build 0000 64-bit Version effective date: March 25, 2019. So far it seems to be working properly for me, though I have had some problems trying to use the PocketSense import function with a Microsoft Money file where I had not previoulsy been using PockeSense.
DeleteYou can read about this situation starting with my post on July 8.
Paul White
The python.org version will not contain the REQUESTS package. See post info at top of this thread for how to install. ActiveState includes the requests package by default.
DeleteRobert - I tried what you did, and the import worked. I tried it with my sites.dat file and it failed, so I did some testing and figured out that I needed to have a SITE entry in sites.dat even though I don't reference the account in Setup. However, I get a +Scrubber message that says Processing Discover Card statement, which is the SITE entry I left in sites.dat. That seems like a bug.
ReplyDeleteThis also seems to work with quotes turned off. Here is what I see when I run Getdata:
PocketSense OFX Download Python Scripts, Ver: 2021-Mar-12 DEV
Download transactions? (Y/N/I=Interactive) [Y] i
Download interval= 45 days
No accounts have been configured. Run SETUP.PY to add accounts
Searching .\import\ for statements to import
Importing 2021-07-11_transaction_download.ofx
+Scrubber: Null time values updated.
+Scrubber: Processing Discover Card statement.
Finished downloading data
Combined OFX created: .\xfr\combined367546.ofx
Upload online data to Money? (Y/N/V=Verify) [Y]
Paul White
Paul White: Regarding the Discover scrubber running, did the file in the import folder by chance contain "discover" (case insensitive)?
ReplyDeleteI see your context switching batch files make good sense.
Cal Learner
***Note/reminder to all: Once there are more than 200, posts in here, you will need to click "Load More ..." to see the lastest posts.
Cal - The file in the import folder does not have the word "discover" in it. This is happening for files from two different accounts. I have no accounts defined in my ofx_config.cfg file, but I have this in my sites.dat file:
Delete<site>
SiteName : DISCOVER
AcctType : CCSTMT #credit card
fiorg : Discover Financial Services
fid : 7101
url : https://ofx.discovercard.com
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval:
timeOffset :
</site%gt;
Maybe this is where Getdata is getting "Processing Discover Card statement"?
I tracked down a bug related to importing files when there are no matching sites in sites.dat, and updating the dev version right after I post this. As for why it's triggering the DISCOVER scrub routine, that should only happen if the file being imported contains a BANKID or FID field that matches the Discover sites.dat entry for the same.
DeleteUSAA problem again. For the past few days, been unable to get downloads. The OFX file that is created shows: "Request unsuccessful. Incapsula incident ID: 531000360002967700-7737735805864197" Anyone else having this issue? Any suggestions?
ReplyDeleteI don't use USAA, but I wonder if the unique client UID has changed (the one you had to get from their site)?
DeleteI use USAA for checking/savings and a credit card and I have had no problems, including today.
DeleteNote that this occurred with both the previous DEV version I had downloaded 13-Mar-2021 and the most recent version I downloaded today. Other sites (Fidelity, Citi, Chase) seem to work OK.
ReplyDelete