wrong accentuated characters

dragonauta

Hi,
I was using Epesi 1.2.2 locally, instaled on my localhost using a LAMP server.
I installed into /www/epesibim folder
I have no problems at all.

I wan't to move it to my web server to make it always available.
I made a folder "epesibim" in my webroot.
Move all the files from my local installation to that web folder
Made a dump (with mysqldump) of database
Made a new database in my web server (same name, assigned same user and password like local)

Also I watched that locally I've created database with collation utf8
So in web server also did collation utf8
made an import of the database dump and run epesi from the web.

It runs ok, except by wrong characters in some parts of the GUI.
In example the "epesi powered version 1.2.2" I see it as "epesi powered versiÃ³n 1.2.2"
note the Ã³ should be an ó character.

Also when I made a new call, or ticket, or an action that requires a client selección
example, I make a new Phone Call, put info and select the client "ITParanAi" (when should be "ITParaná") and save the call.
Then I see ITParaná (no wrong chars)

The strange is that If I go to Companies or Contacts, I see right characters: á, é, í, ó, ú, ñ
The bug appear when I make a new event, call, task... when write part of client's name and the yellow list drops down to select one.

help appretiated
thanks!

jasiek

Hi,
It looks like encoding is not correct. Please see if your database is setup for UFT-8 and then the second option is to select collocation: we use utf8_unicode_ci
See if you can setup the database like this and import again.

dragonauta

Sorry, we encountered an error while displaying this content. If you're a user, please try again later. If you're an administrator, take a look in your Flarum log files for more information.

dragonauta

made a full backup,
delete whole database
create a new one, taking care about collating = utf8_unicode_ci
erase all files from data folder except index.html
launch again epesi and got the setup.

check into phpmyadmin and database and tables are utf8_unicode_ci
when launch re-installed epesi, I see same odd chars... I think it's my hosting.

ethnar

Hello,

The issue with the encoding seem to be unrelated to the database collation you've set. The database might be (and by what you write, it is) properly configured. The reason why I'm thinking so is that you mention the problem with string "epesi powered versión 1.2.2". Fact is, that string does not interact with the database, it's hardcoded.
You also mentioned that when the tooltip for version check loads, that's when the "version" string becomes "corrupted". It'd suggest that the ajax contents sent to the browser is not encoded correctly.

It is very likely there's some misconfiguration on the hosting provider end. However, to be perfectly sure I'd remove the browser from the equation: please try to access the site using different browser, or better yet, from different machine. If the error persists, it's a problem related to hosting provider. It's really hard for me to suggest any steps how to work around this bug, I've never experienced anything like it.

Kind regards,
Arek

dragonauta

Thank for your reply
Maybe I focused very much on the epesi powered text turning this bug into trivial,
but also I see those chars in the query results when looking for a client o company.

When creating new event: phone call, task, ticket, project, etc, etc
Type part of client/company's name and get a list
(I assume it's an ajax query on database, correct me if I'm wrong.)
This (yellow) list shows wrong encoding. (attached a PNG for reference.)

But then, when I save the event, the encoding back to normal and I see no wrong chars.

About browsers:
I've tried Chrome and FF in my Ubuntu PC, both shows wrong encoding.
Also looking at page's source code I see charset utf8
I've tried in FF and IE9 from a W7 notebook, exactly same results as before in Ubuntu.

Maybe it's a hosting misconfiguration... but also seems like ajax is messing around with encoding in some parts.

For reference, in that same hosting provider (same account) I have running different services: a wordpress site, a dokuwiki installation and a fully working phplist, all of them configured for spanish and none of them present this issue with encodings.

Since I have the issue and couldn't be reproduced, should I try something? where should I look for?

thank you very much for your precious time

wrong chars when lookin for client

ethnar

The fact that company names are displayed properly once the record is saved confirms the issue is not related to database.

I think the easiest way to go about debugging it is to stick with the version check. The code there is much simpler and easier to understand than autocomplete and I'm almost certain the underlying issue is the same for both cases.

The version check is triggered from the Base/Box module. All the files related can be found in modules/Base/Box. Box_0.php has code that simply triggers the javascript function check_for_new_version(), you don't even have to open/read the Box_0.php.
The function is placed in the check_for_new_version.js file (found in the same folder). That javascript function creates new ajax request to check_for_new_version.php - again, file you can find the same folder. This is where you should start. Check what this script generates (using functions like file_put_contents()), check if the encoding is proper in those files, make sure you display them with utf8-compatible editor. If they are fine, check what your browser gets as a response for this request. After that it's all up to Javascript to place it on the page.

That's all I can suggest for now. If you have any further questions, don't hesitate to ask. I'd be very much interested in knowing what causes an issue like that.

Kind regards,
Arek

dragonauta

Using Firefox 11.0 for Ubuntu:
- open the web console
- Start Epesi and watch console
- I see good encoding until check_for_new_version.php finish loading, then bug happens.
[17:05:48.136] POST http://xxxxxxx.com.ar/epesi/modules/Bas ... ersion.php [HTTP/1.1 200 OK 3324ms]

- also source code on browser returns: <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />, but looking at Character Encoding menu returns ISO-8859-1

A curiosity:
I'm asking this cause I'm a little ignorant in this issue:
Is there any chance that a file's charset could change the encoding of what I'm seeing?

I detected that some es.php generated files by translation are us-ascii encoded, some other are utf8
I used this one-liner to obtain charset: for F in `ls`; do file -i $F; done
by example:
into modules/Utils/Tooltip/lang/ I see: $
[list:3ibishik]de.php: text/x-php; charset=us-ascii
en.php: text/x-php; charset=us-ascii
es.php: text/x-php; charset=us-ascii
et.php: text/x-php; charset=us-ascii
fr.php: text/x-php; charset=us-ascii
pl.php: text/x-php; charset=us-ascii
ro.php: text/x-php; charset=us-ascii
ru.php: text/x-php; charset=utf-8
sc.php: text/x-php; charset=utf-8[/list:u:3ibishik]

into modules/CRM/Contacts/lang/ I see:
[list:3ibishik]cz.php: text/x-php; charset=utf-8
de.php: text/x-php; charset=utf-8
en.php: text/x-php; charset=us-ascii
es.php: text/x-php; charset=utf-8
et.php: text/x-php; charset=utf-8
fr.php: text/x-php; charset=us-ascii
lt.php: text/x-php; charset=utf-8
nl.php: text/x-php; charset=us-ascii
pl.php: text/x-php; charset=utf-8
ro.php: text/x-php; charset=us-ascii
ru_n.php: text/x-php; charset=utf-8
ru.php: text/x-php; charset=utf-8
sc.php: text/x-php; charset=utf-8
sv.php: text/x-php; charset=utf-8
[/list:u:3ibishik]

ethnar

Hello,

The encoding of files does matter, but from your description it's not the issue here, because this also affects Company names returned with ajax requests and these do not interact with file contents. You may want to check the file data/Base_Lang/base/es.php if the strings translated there are properly encoded, but I doubt you'll find any problems there.

Can you post here your request header and response body? For instance, mine are:

Request URL:http://localhost/epesi/modules/Base/Box/check_for_new_version.php

Request Method:POST

Status Code:200 OK

Request Headersview source

Accept:text/javascript, text/html, application/xml, text/xml, */*

Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3

Accept-Encoding:gzip,deflate,sdch

Accept-Language:en-US,en;q=0.8,pl;q=0.6

Connection:keep-alive

Content-Length:5

Content-type:application/x-www-form-urlencoded; charset=UTF-8

Cookie:PHPSESSID=u3mbpg26kve1ticu3hbv4q0jm3

Host:localhost

Origin:http://localhost

Referer:http://localhost/epesi/

User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.83 Safari/535.11

X-Client-ID:0

X-Prototype-Version:1.7

X-Requested-With:XMLHttpRequest

Form Dataview URL encoded

cid:0

Response Headersview source

Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Connection:Keep-Alive

Content-Length:350

Content-Type:text/html

Date:Tue, 03 Apr 2012 12:42:11 GMT

Expires:Thu, 19 Nov 1981 08:52:00 GMT

Keep-Alive:timeout=5, max=100

Pragma:no-cache

Server:Apache/2.2.21 (Win32) mod_ssl/2.2.21 OpenSSL/1.0.0e PHP/5.3.8 mod_perl/2.0.4 Perl/v5.10.1

X-Powered-By:PHP/5.3.8

And the response:

<span  onMouseMove="if(typeof(Utils_Tooltip__showTip)!='undefined')Utils_Tooltip__showTip(this,event,300)" tip="You are using most up-to-date version of EPESI." onMouseOut="if(typeof(Utils_Tooltip__hideTip)!='undefined')Utils_Tooltip__hideTip()" onMouseUp="if(typeof(Utils_Tooltip__hideTip)!='undefined')Utils_Tooltip__hideTip()" >versión 1.3</span>

My suggestion from earlier post to investigate all stages of preparing the reply is still valid, that's where I'd start.

Kind regards,
Arek

dragonauta

Ok. tried locally. No problem. Updated to 1.3 version. No problem

Here's the request and response from my localhost

URL de la petición: http://webserver/epesi/modules/Base/Box/check_for_new_version.php Método de la petición: POST Código de estado: HTTP/1.1 200 OK Cabeceras de la petición Accept:text/javascript, text/html, application/xml, text/xml, */* Accept-Encoding:gzip, deflate Accept-Language:es-es,es;q=0.8,en-us;q=0.5,en;q=0.3 Cache-Control:no-cache Connection:keep-alive Content-Length:5 Content-Type:application/x-www-form-urlencoded; charset=UTF-8 DNT:1Host:webserver Pragma:no-cache Referer:http://webserver/epesi/index.php User-Agent:Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0 X-Client-ID:2 X-Prototype-Version:1.7 X-Requested-With:XMLHttpRequest

Cabeceras de la respuesta Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Connection:Keep-Alive Content-Encoding:gzip Content-Length:20 Content-Type:text/html Date:Wed, 04 Apr 2012 19:46:39 GMT Expires:Thu, 19 Nov 1981 08:52:00 GMT Keep-Alive:timeout=5, max=100 Pragma:no-cache Server:Apache/2.2.20 (Ubuntu) Vary:Accept-Encoding X-Powered-By:PHP/5.3.6-13ubuntu3.6

Then I exported everything to my webhost and here's the request and response:

URL de la petición: http://itparana.com.ar/epesi/modules/Base/Box/check_for_new_version.php Método de la petición: POST Código de estado: HTTP/1.1 200 OK Cabeceras de la petición Accept:text/javascript, text/html, application/xml, text/xml, */* Accept-Encoding:gzip, deflate Accept-Language:es-es,es;q=0.8,en-us;q=0.5,en;q=0.3 Cache-Control:no-cache Connection:keep-alive Content-Length:5 Content-Type:application/x-www-form-urlencoded; charset=UTF-8 DNT:1Host:itparana.com.ar Pragma:no-cache Referer:http://itparana.com.ar/epesi/ User-Agent:Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0 X-Client-ID:2 X-Prototype-Version:1.7 X-Requested-With:XMLHttpRequest

Cabeceras de la respuesta Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Connection:close Content-Encoding:gzip Content-Length:20 Content-Type:text/html; charset=ISO-8859-1 Date:Wed, 04 Apr 2012 19:41:32 GMT Expires:Thu, 19 Nov 1981 08:52:00 GMT Pragma:no-cache Server:Apache Vary:Accept-Encoding,User-Agent

Seems that my webhost send ISO-8859-1 charset instead UTF-8

Also I have a Phone Call, and pointing to the contact I see an AJAX box with data
So I look into console and see that this file is loaded
[quote:2h1gikf3]http://itparana.com.ar/epesi/modules/Utils/Tooltip/req.php[/quote:2h1gikf3]
and also the request is UTF-8 and the response returns ISO-8859-1

ethnar

Hello,

So it seems it's hosting issue. It'd very odd to have responses encoded in charset=ISO-8859-1 by default.

What's important now is to try to work around this. I've found this article and seems very likely one of the solutions provided there should fix it:
http://www.askapache.com/htaccess/setti ... ccess.html
Just make sure there are no .htaccess files in /modules/, /modules/Base/ and /modules/Base/Box/ - then you only need to edit the .htaccess in the main folder. Of course the <meta> solution is off the table, since the problematic responses are usually javascript. This is also the reason why your EPESI works in general (we include that tag by default) and ajax requests are the only one that have the problem.

If it works, I should point out that there's another .htaccess in /data/ folder and it'll be safer to set this parameter there as well.
Please tell us if one of these solutions fixed the issue.

Kind regards,
Arek

dragonauta

Here are changes I made (still get wrong encoding):

My .htaccess into epesi folder

Options -Indexes

SetEnv PHPRC /www/itparana.com.ar/htdocs/epesi

AddDefaultCharset UTF-8

my php.ini, into epesi folder

magic_quotes_gpc = Off

memory_limit = "64M"

default_charset = "utf-8"

A curious warning when looking at firefox console:
[15:26:55.834] No se reconoce la regla at o error al leer regla at '@CHARSET'. @ http://itparana.com.ar/epesi/data/Base_ ... /__css.php

In english: Rule is not recognized at or error reading rule at '@ CHARSET'

So I double clic and bring me this:
[snip]
table{width:100%}@CHARSET "ISO-8859-1";.no_border{border:none}hr.small{background:#000;height:1px}p.event{font-size:7px;line-height:11px;margin:4px
[snip]

Can it be that "charset" in "css" produces this problem with the encoding?

Also I contacted my webhost provider informing this issue about encoding (that request and response are different)

ethnar

CSS in general doesn't require the use of utf-8 encoding, there's no problem there. This encoding set there should affect only that one request - I think you're getting this warning for the very same reason you get the problem with ajax requests.

Did you try the other solutions suggested in the article?
We should also check if your webhost even allows to change certain settings via .htaccess at all. To do that, the simplest way is to use a certain .htaccess file and check if the server response is appropriate.
Put this code in your .htaccess file:

#Check if htaccess is used

Wrong Parameter On Purpose

And try to run the site. If it crashes with an server error 500, that's alright - it means that .htaccess is being parsed and used. It also means that without that line, the .htaccess is interpreted properly, without errors and that means any directives you put there are in effect. Which would mean that if you do try the other solutions suggested and they don't work, we're in sort of dead-end.

I won't dwell on that right now thou, because it's actually very likely the hosting you're using has set AllowOverride None and trying to launch the site with invalid .htaccess file still brings up the site itself, without error. If that's the case, you should speak with your hosting provider and, I dare say, demand they set to allow to override certain settings through .htaccess, at the very least FileInfo that's needed to force utf-8 encoding through the method suggested in article I linked.

If you have any questions don't hesitate to ask.

Kind regards,
Arek

dragonauta

Sorry for that, I missed to comment that I've tried every solution provided by your link.
Also tried to force UTF8 by php.ini into epesi folder. (just like my previous post)

About new directives:
I put the codes into .htaccess and gave me 500 error. So .htacces file is being parsed correctly.
Looks like it's a dead end.

--------------------
Another test I made:
called epesi from firefox, I get ISO-8859-1 encoding.
Go to menu -> Web development -> Encoding -> set UTF-8
The page reloads and BAM... white page.

just 2 lines in console :

[22:25:42.114] POST http://itparana.com.ar/epesi/process.php [HTTP/1.1 200 OK 5217ms]

[22:25:47.335] shortcut is not defined @ http://itparana.com.ar/epesi/#0

The request asks for utf-8 and response says:
[quote:1beffvs3]Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:close
Content-Encoding:gzip
Content-Length:416
Content-Type:text/javascript
Date:Fri, 06 Apr 2012 01:25:46 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Pragma:no-cache
Server:Apache
Vary:Accept-Encoding,User-Agent[/quote:1beffvs3]

I'm opening a ticket at my hosting provider to see if we can solve this issue.
thanks

dragonauta

Hi!
I was looking at the TooltipCommon0.php code, and explore every function, using google.
Then I found it:

Could be ... ( and please consider that I'm not a php developer )
... that php function htmlspecialchars throws ISO-8859-1 by default instead UTF-8?

My host is running PHP 5.3.6
I read that on php 5.4 they change htmlspecialchars' default to UTF-8

Also, I went through code seeking htmlspecialchars function and sometimes it has just the text argument and sometimes specifying UTF-8.

Can we override this function to force UTF-8 if no parameter is specified? how?
I think it would be an interesting test.
thanks

dragonauta

Ok, made a test.
search all references of htmlspecialchars and added ENT_QUOTES,'UTF-8'
even if not needed.
Then enter into admin and rebuild theme and cache.
Enter epesi and try, still see the issue.

But I identified that wrong encoding appears here:
- only in phonecall applet, when cursor is over contact, the ajax box displays wrong encoding
- when creating a phonecall, meeting (or any action) when write some characters to seek a client I see wrong encoding into results box
- I have 1.3 version, so the epesi powered version changed to "epesi powered", I don't see anymore "version X.X.X" (in version 1.2.2 I saw wrong encoding)

The rest of application runs very well, no wrong encodings.
I explored all tables in database, no wrong characters saved into records.
I can't figure what is going wrong. :?

ethnar

Hello,

The htmlspecial chars is widely used in EPESI and if that would be the reason, you'd see the issue appear in multiple places.

The reason version now displays properly is because we changed the way EPESI checks for updates. And for the new check to work, your EPESI installation must be registered. In the release there's a minor bug where unregistered version may see the text "version 1.3" disappear. Once registered, it should run normally.

I believe the issue is strictly linked to some very peculiar configuration on the hosting side. I'm not sure if there's anything that can be done on the EPESI side to work around that, there's some chance for that since the module contents is also loaded with ajax. I'm afraid it's too complicated for me to give any solid guidelines. If you want, I can take a look at an example installation on this hosting, one that suffers from the issue you mentioned.

Kind regards,
Arek

dragonauta

Thank you so much for your help. I'm giving up.

This issue doesn't affect my work and it's not that bad (just annoying in some parts).
As I said before, epesi runs flawlessly locally under php 5.3.6 (curiously same version as my hosting provider), so looks like it is a peculiar [mis]configuration (?)

What I did:
[*] added/tried default charset using .htaccess and epesi/php.ini files.
[*] I've searched and added in every htmlspecialchars mention the UTF-8 encoding (about 170 lines of code in different files :lol: )
[*] also changed all @CHARSET ISO-8859-1 into css files to UTF-8 (about 10 files)
[*] I rebuilt cache and theme under epesi/admin/

None of those actions results as expected... keep getting ISO-8859-1 charset.
I repeat: only affects on some parts (some tooltips and data retrieve like typing contact search) but data is saved ok.
So, it's not a big deal.
I can live with those odd chars :roll:

Thanks again!

Epesi ENS | Epesi Academy | epesi.cloud PaaS | GitHub

Help Me! | Download from SourceForge