Asterisk或者FreePBX出现故障如何排查是工程师最基本的技能之一。如何排查复杂的底层问题是技术人员经常遇到的问题，如果技术人员能够提供非常完整的debug消息，这样就会大大提高排查问题的速度，提高工作效率可以快速和其他支持人员沟通。
在Asterisk和FreePBX环境下，官方提供了多种排查的方式，通过价值排查方式可以轻松解决这些问题，用户可以排查应用级FreePBX和界面的问题，也可以排查Asterisk平台级的问题（系统崩溃，死锁等）。 主要的排查方式包括：
- FreePBX CLI Debug
- FreePBX Debug
- Browser Debug （Console Log）
- CHROME
- SAFARI
- IE9:
- FIREFOX
- OPERA
- Asterisk Logs
Backtraces （Segfaults/Core Dumps/Asterisk Crashing）
1）通过执行命令来激活Debug方式，保存相关的日志到相应的默认路径。 在linux CLI环境下，执行amportal a dbug，显示以下输出结果， 开发人员可以根据相应的路径文件检查问题。
[root@dahdidev1 ~]# amportal a dbug
Fetching FreePBX settings with gen_amp_conf.php
trying to run as user asterisk:
[FPBXDBUGDISABLE] already set to [0]
==> /var/log/asterisk/freepbx_dbug <==
==> /var/log/httpd/error_log <==
mv: cannot stat `/var/lib/asterisk/agi-bin/license-*.zl': No such file or directory
[Wed Dec 03 00:05:09 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/ucp/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/restapi
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /tftpboot/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/aastra
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/restapps
[Wed Dec 03 01:59:48 2014] [error] [client 94.102.49.82] script '/var/www/html/recordings/index.php' not found or unable to stat
[Wed Dec 03 01:59:50 2014] [error] [client 94.102.49.82] File does not exist: /var/www/html/admin/modules/admindashboard
[Wed Dec 03 01:59:51 2014] [error] [client 94.102.49.82] client denied by server configuration: /var/www/html/admin/page.framework.php
==> /var/log/asterisk/freepbx.log <==
[2014-Dec-03 09:44:02] [PHP-WARNING] （/var/www/html/admin/modules/endpoint/functions.inc/functions_epm_ucp.php:0） - No license for this product （PBXact） - make sure zend_loader.license_path is properly configured in your ini file!
[2014-Dec-03 09:44:02] [PHP-WARNING] （/var/www/html/admin/modules/endpoint/functions.inc/functions_epm_ucp.php:0） - License check failed!
[2014-Dec-03 09:57:01] [WARNING] （libraries/modulefunctions.legacy.php:7） - Depreciated Function _module_distro_id detected in /var/www/html/admin/modules/sysadmin/functions.inc/general.php on line 107
[2014-Dec-03 09:57:01] [PHP-NOTICE] （/var/www/html/admin/modules/sysadmin/functions.inc/license.php:53） - Undefined index: register
[2014-Dec-03 09:57:02] [WARNING] （libraries/modulefunctions.legacy.php:7） - Depreciated Function _module_distro_id detected in /var/www/html/admin/modules/sysadmin/functions.inc/general.php on line 107
[2014-Dec-03 09:57:02] [PHP-NOTICE] （/var/www/html/admin/modules/sysadmin/functions.inc/license.php:53） - Undefined index: register
[2014-Dec-03 09:58:20] [INFO] （bin/freepbx_setting:39） - trying to run as user asterisk:
[2014-Dec-03 09:58:20] [INFO] （bin/freepbx_setting:40） -
[2014-Dec-03 09:58:21] [INFO] （bin/freepbx_setting:61） - [FPBXDBUGDISABLE] already set to [0]
[2014-Dec-03 09:58:21] [INFO] （bin/freepbx_setting:62） -
2）Browser Debug （Console Log），这个方法是通过浏览器的开放模式来检查问题。开发人员需要根据自己所使用的浏览器，通过不同的命令方式来排查。
CHROME:
- Press either CTRL + SHIFT + J to open the “console” tab of the Developer Tools.
- Alternative method:
- Press either CTRL + SHIFT + I or F12 to open the Developer Tools.
- Press ESC （or click on “Show console” in the bottom right corner） to slide the console up.
- Note: In Chrome’s dev tools, there is a “console” tab. However, a smaller “slide-up” console can be opened while any of the other tabs is active.
SAFARI:
- Press CTRL + ALT + I to open the Web Inspector.
- See Chrome’s step 2. （Chrome and Safari have pretty much identical dev tools.）
- Note: Step 1 only works if the “Show Develop menu in menu bar” check box in the Advanced tab of the Preferences menu is checked!
IE9:
- Press F12 to open the developer tools.
- Click the “console” tab.
FIREFOX:
Press CTRL + SHIFT + K to open the Web console （COMMAND + SHIFT + K on Macs）。
or, if Firebug is installed:
Press F12 to open Firebug.
Click on the “console” tab.
OPERA:
Press CTRL + SHIFT + I to open Dragonfly.
Click on the “console” tab.
3） Asterisk Logs 日志文件
root@freepbxdev1 ivr]# tail -f /var/log/asterisk/full // 默认asterisk日志存储路径
[2014-12-03 13:20:53] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:22:41] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:23:22] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:25:51] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:27:41] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:27:59] WARNING[6287] res_calendar_caldav.c: Unknown response to CalDAV calendar myGoogleCal, request REPORT to /calendar/dav/en.usa%23holiday%40group.v.calendar.google.com/events: SSL handshake failed: SSL error: GnuTLS internal error.
[2014-12-03 13:28:20] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:30:49] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:32:41] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:33:18] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
如果需要查找特定的条件的话，也可以通过以下命令grep排查报错日志信息：
[root@34693894 ~]# grep 1518526777.67 /var/log/asterisk/full*
- var/log/asterisk/full-20180214:[2018-02-13 08:59:37] VERBOSE[24184][C-00000001] pbx.c: Executing [s@macro-user-callerid:1] Set（"SIP/5002-00000001", "TOUCH_MONITOR=1518526777.67"） in new stack
- var/log/asterisk/full-20180214:[2018-02-13 08:59:37] VERBOSE[24184][C-00000001] pbx.c: Executing [700@from-internal:37] QueueLog（"SIP/5002-00000001", "700,1518526777.67,NONE,DID,"） in new stack
4 可以使用命令查找方式：
ls -l /tmp | grep core. // 找到相关的core dump 信息。
5 可以使用Sangoma 的开发排查根据来排查，用户需要安装这些开发工具，安装的是ISO版本：
# 仅支持 FreePBX Distro 7!
yum install -y sangoma-devel
debuginfo-install --enablerepo=centos7-debuginfo asterisk14 // 这里的支持的是asterisk 14，用户可以使用不同的版本，例如15来排查Asterisk15.
6 如果用户运行 13.14.0 或者 14.3.0以上版本，也可以使用系统默认支持的工具：
/var/lib/asterisk/scripts/ast_coredumper /tmp/[name of the core file]
7 使用最简单命令获得崩溃信息：
gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c [name of the core file] > /tmp/backtrace.txt
8 Asterisk新排查利器：scripts/ast_coredumper 命令
此命令是13.14.0 和 14.3.0以上版本添加到Asterisk崩溃系统命令，功能非常强大，可以轻松排查各种asteisk问题，死锁，收集系统日志信息。配置文件在默认Asterisk路径：etc/asterisk/ast_debug_tools.conf。具体命令使用方式如下：
var/lib/asterisk/scripts/ast_coredumper --help
AME
ast_coredumper - Dump and/or format asterisk coredump files
SYNOPSIS
ast_coredumper [ --help ] [ --running | --RUNNING ] [ --latest ]
[ --tarball-coredumps ] [ --delete-coredumps-after ]
[ --tarball-results ] [ --delete-results-after ]
[ --tarball-uniqueid="<uniqueid>" ]
[ --no-default-search ] [ --append-coredumps ]
[ <coredump> | <pattern> … ]
DESCRIPTION
Extracts backtraces and lock tables from Asterisk coredump files.
For each coredump found, 4 new result files are created:
- <coredump>.brief.txt: The output of "thread apply all bt".
- <coredump>.thread1.txt: The output of "thread apply 1 bt full".
- <coredump>.full.txt: The output of "thread apply all bt full".
- <coredump>.locks.txt: If asterisk was compiled with
"DEBUG_THREADS", this file will contain a dump of the locks
table similar to doing a "core show locks" from the asterisk
CLI.
Optional features:
- The running asterisk process can be suspended and dumped.
- The coredumps can be merged into a tarball.
- The coredumps can be deleted after processing.
- The results files can be merged into a tarball.
- The results files can be deleted after processing.
Options:
--help
Print this help.
--running
Create a coredump from the running asterisk instance and
process it along with any other coredumps found （if any）。
WARNING: This WILL interrupt call processing. You will be
asked to confirm.
--RUNNING
Same as --running but without the confirmation prompt.
DANGEROUS!!
--latest
Process only the latest coredump from those specified （based
on last-modified time）。 If a dump of the running process was
requested, it is always included in addition to the latest
from the existing coredumps.
--tarball-coredumps
Creates a gzipped tarball of all coredumps processed.
The tarball name will be:
/tmp/asterisk.<timestamp>.coredumps.tar.gz
--delete-coredumps-after
Deletes all processed coredumps regardless of whether
a tarball was created.
--tarball-results
Creates a gzipped tarball of all result files produced.
The tarball name will be:
/tmp/asterisk.<timestamp>.results.tar.gz
--delete-results-after
Deletes all processed results regardless of whether
a tarball was created. It probably doesn't make sense
to use this option unless you have also specified
--tarball-results.
--tarball-uniqueid="<uniqueid>"
Normally DATEFORMAT is used to make the tarballs unique
but you can use your own unique id in the tarball names
such as the Jira issue id.
--no-default-search
Ignore COREDUMPS from the config files and process only
coredumps listed on the command line （if any） and/or
the running asterisk instance （if requested）。
--append-coredumps
Append any coredumps specified on the command line to the
config file specified ones instead of overriding them.
<coredump> | <pattern>
A list of coredumps or coredump search patterns. Unless
--append-coredumps was specified, these entries will override
those specified in the config files.
Any resulting file that isn't actually a coredump is silently
ignored. If your patterns contains spaces be sure to only
quote the portion of the pattern that DOESN'T contain wildcard
expressions. If you quote the whole pattern, it won't be
expanded.
If --no-default-search is specified and no files are specified
on the command line, then the only the running asterisk process
will be dumped （if requested）。 Otherwise if no files are
specified on the command line the value of COREDUMPS from
ast_debug_tools.conf will be used. Failing that, the following
patterns will be used:
/tmp/core[-._]asterisk!（*.txt）
/tmp/core[-._]$（hostname）！（*.txt）
NOTES
You must be root to use ast_coredumper.
The script relies on not only bash, but also recent GNU date and
gdb with python support. *BSD operating systems may require
installation of the 'coreutils' and 'devel/gdb' packagess and minor
tweaking of the ast_debug_tools.conf file.
Any files output will have ':' characters changed to '-'. This is
to facilitate uploading those files to Jira which doesn't like the
colons.
FILES
/etc/asterisk/ast_debug_tools.conf
~/ast_debug_tools.conf
./ast_debug_tools.conf
#
# This file is used by the Asterisk debug tools.
# Unlike other Asterisk config files, this one is
# "sourced" by bash and must adhere to bash semantics.
#
# A list of coredumps and/or coredump search patterns.
# Bash extended globs are enabled and any resulting files
# that aren't actually coredumps are silently ignored
# so you can be liberal with the globs.
#
# If your patterns contains spaces be sure to only quote
# the portion of the pattern that DOESN'T contain wildcard
# expressions. If you quote the whole pattern, it won't
# be expanded and the glob characters will be treated as
# literals.
#
# The exclusion of files ending ".txt" is just for
# demonstration purposes as non-coredumps will be ignored
# anyway.
COREDUMPS=（/tmp/core[-._]asterisk!（*.txt） /tmp/core[-._]$（hostname）！（*.txt））
# Date command for the "running" coredump and tarballs.
# DATEFORMAT will be executed to get the timestamp.
# Don't put quotes around the format string or they'll be
# treated as literal characters. Also be aware of colons
# in the output as you can't upload files with colons in
# the name to Jira.
#
# Unix timestamp
#DATEFORMAT='date +%s.%N'
#
# *BSD/MacOS doesn't support %N but after installing GNU
# coreutils…
#DATEFORMAT='gdate +%s.%N'
#
# Readable GMT
#DATEFORMAT='date -u +%FT%H-%M-%S%z'
#
# Readable Local time
DATEFORMAT='date +%FT%H-%M-%S%z'
以上命令需要asterisk的相关源代码编译参数，例如 DEBUG_THREADS等相关参数。
通过以上几种排查方式，技术人员可以轻松排查各种IPPBX界面以及应用方面的问题，也可以轻松排查Asterisk底层系统的崩溃，死锁问题。特别是Asterisk系统底层的工具给开发人员带来了极大的便利。
参考资料：
https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace
http://freepbx.org.cn/wiki/index.php?title=FreePBX
关注微信公众号：asterisk-cn，获得有价值的Asterisk行业分享
Asterisk freepbx 中文官方论坛：http://bbs.freepbx.cn/forum.php
Asterisk freepbx技术文档： www.freepbx.org.cn
融合通信商业解决方案，协同解决方案首选产品：www.hiastar.com
Asterisk/FreePBX中国合作伙伴，官方qq技术分享群（3000千人）：589995817
